Limited social TV apparatus

ABSTRACT

One aspect of the disclosed technology is a limited social television system that provides a light-weight process that allows multiple audience members to close the loop of interaction and create a compelling feeling of “shared experience” on the part of the response-reviewer.

CROSS-REFERENCE TO RELATED APPLICATIONS

patent application Ser. No. ______, filed herewith on ______, entitledMethods, Apparatus, and Program Products to Close Interaction Loops forSocial TV, Attorney Docket Number 20050596-US-NP, is hereby incorporatedby reference in its entirety. The inventors of the instant applicationhave assigned, or are subject to an obligation of assignment of theinstant application to the assignee of application ______.

Published U.S. Patent Application No: US 2004/0172255 A1, filed Apr. 16,2003, entitled Methods, Apparatus, and Products for AutomaticallyManaging Conversational Floors in Computer-Mediated Communications, byAoki et al. is hereby incorporated by reference in its entirety. Theinventors of the instant application have assigned, or are subject to anobligation of assignment of the instant application to the assignee ofapplication Ser. No. 10/414,923.

Published U.S. Patent Application No: US 2004/0172252 A1, filed Apr. 16,2003, entitled Methods, Apparatus, and Products for Identifying aConversation, by Aoki et al. is hereby incorporated by reference in itsentirety. The inventors of the instant application have assigned, or aresubject to an obligation of assignment of the instant application to theassignee of application Ser. No. 10/414,912.

Published U.S. Patent Application No: US 2003/0056220 A1, filed Sep. 14,2001, entitled System and Method for Sharing and Controlling MultipleAudio and Video Streams, by Thornton et al. is hereby incorporated byreference in its entirety. The inventors of the instant application haveassigned, or are subject to an obligation of assignment of the instantapplication to the assignee of application Ser. No. 09/951,726.

Patent application Ser. No. 10/997,216, filed Oct. 29, 2004, entitledMethods, Apparatus, and Program Products to Support a Shared ViewingExperience from Remote Locations, is hereby incorporated by reference inits entirety. The inventors of the instant application have assigned, orare subject to an obligation of assignment of the instant application tothe assignee of application Ser. No. 10/997,216.

Patent application Ser. No. 10/997,301, filed Oct. 29, 2004, entitledMethods, Apparatus, and Program Products for Presenting ReplacementContent Instead of a Portion of a Recorded Content, is herebyincorporated by reference in its entirety. The inventors of the instantapplication have assigned, or are subject to an obligation of assignmentof the instant application to the assignee of application Ser. No.10/997,301.

Patent application Ser. No. 10/997,215, filed Oct. 29, 2004, entitledMethods, Apparatus, and Program Products for Presenting Commentary Audiowith Recorded Content, is hereby incorporated by reference in itsentirety. The inventors of the instant application have assigned, or aresubject to an obligation of assignment of the instant application to theassignee of application Ser. No. 10/997,215.

Patent application Ser. No. 10/997,302, filed Oct. 29, 2004, entitledMethod for Providing Commentary Audio and Additional or ReplacementVideo Content, is hereby incorporated by reference in its entirety. Theinventors of the instant application have assigned, or are subject to anobligation of assignment of the instant application to the assignee ofapplication Ser. No. 10/997,302.

Patent application Ser. No. 10/996,896, filed Oct. 29, 2004, entitledMethod, Apparatus, and Program Products for Socially Synchronizing anExperiential Data Stream, is hereby incorporated by reference in itsentirety. The inventors of the instant application have assigned, or aresubject to an obligation of assignment of the instant application to theassignee of application Ser. No. 10/996,896.

Patent application Ser. No. 10/997,188, filed Oct. 29, 2004, entitledMethod and Apparatus for Controlling an Experiential Data Stream in aSocial Space, is hereby incorporated by reference in its entirety. Theinventors of the instant application have assigned, or are subject to anobligation of assignment of the instant application to the assignee ofapplication Ser. No. 10/997,188.

Patent application Ser. No. 10/997,189, filed Oct. 29, 2004, entitledMethods, Apparatus, and Program Products for Aligning Presentation ofSeparately Recorded Experiential Data Streams, is hereby incorporated byreference in its entirety. The inventors of the instant application haveassigned, or are subject to an obligation of assignment of the instantapplication to the assignee of application Ser. No. 10/997,189.

Patent application Ser. No. 10/997,292, filed Oct. 29, 2004, entitledMethod and Apparatus for Associating Commentary Audio with a Position inan Experiential Data Stream, is hereby incorporated by reference in itsentirety. The inventors of the instant application have assigned, or aresubject to an obligation of assignment of the instant application to theassignee of application Ser. No. 10/997,292.

Patent application Ser. No. 10/997,274, filed Oct. 29, 2004, entitledMethods, Apparatus, and Program Products for Controlling Presentation ofan Experiential Data Stream Responsive to Conversations in a SharedSocial Space, is hereby incorporated by reference in its entirety. Theinventors of the instant application have assigned, or are subject to anobligation of assignment of the instant application to the assignee ofapplication Ser. No. 10/997,274.

Patent application Ser. No. 10/997,214, filed Oct. 29, 2004, entitledMethods, Apparatus, and Program Products for Providing SupplementalContent to a Recorded Experiential Data Stream, is hereby incorporatedby reference in its entirety. The inventors of the instant applicationhave assigned, or are subject to an obligation of assignment of theinstant application to the assignee of application Ser. No. 10/997,214.

BACKGROUND

1. Technical Field

The disclosed technology and embodiments relate to the field of creatingsocial spaces for sharing experiences from watching and/or listening torecorded audio or audio/visual information between remotely-locatedaudience members.

2. Background Art

People socialize within a shared social space. Often this socializationoccurs in the context of listening to music, watching a televisionprogram, or watching a movie within the home. Family traditions can becreated, for example, by the sharing of popcorn around a favoritetelevision program on a Sunday night. Often, a favorite movie on VCR orDVD can be watched. The fellowship engendered by this experience changeswhen some of the members of the social group move away or otherwisebecome remote from the heart of the group. This can happen, for example,when the group is a family and a child leaves to go to college. It canalso happen when a group of friends graduates from college or highschool. Often the person who is no longer able to be present misses thegroup and vice versa.

FIG. 1 illustrates a first representation of a prior art social space100 showing a program time axis 101 extending from a program start point103 to a program stop time 105. In this example, a first audience member107 and a second audience member 109 are sharing the experience ofwatching the program in real time while the experiential data streamrepresenting the program is being recorded as it is received at the samelocation (thus, local audience members—the viewers or listeners—areco-present). The playback position/playback vector 111 indicates thelocal audience member's playback position in the experiential datastream and the playback vector (direction and velocity). In this figure,an available content portion 113 indicates the amount of theexperiential data stream that has been received and recorded while anunavailable content portion 115 indicates the amount of the experientialdata stream that has not yet been received. In this figure, the localaudience members experience the audiovisual representation of theexperiential data stream as the data is received. This figurecorresponds to two people in the same location watching a broadcasttelevision program (or listening to an audio program) while recordingthe broadcast material on an audio or video tape or by use of a digitalvideo recorder. Each local audience member can comment about, or reactto, the program to the other local audience member as the program isbeing experienced. While the home is an example setting of this sharedsocial space, other examples include Sports bars, local multi-playervideo games, audio/visual rooms in rest homes, community centers, dormrooms, et cetera.

FIG. 2 illustrates a representation of separate prior art social spaces200 showing a first program experience 201 and a delayed first programexperience 203. The first program experience 201 has a first programstart point 205 and the delayed first program experience 203 has adelayed first program start point 207. A first audience member 209 and asecond audience member 211 are in a shared social space, and in the casewhere the experiential data stream is a video program, watching theexperiential data stream at a first playback position/playback vector213. A third audience member 215 and a fourth audience member 217 arewatching the experiential data stream at a second playbackposition/playback vector 219. The figure also includes a real-time axis221. The situation shown in FIG. 2 often happens, for example, when thedelayed first program experience 203 and the first program experience201 are provided over different networks (for example one provided overa broadcast network and the other provided over a cable network). Thissituation results in a program displacement 223 that can be small orlarge depending on the networks. The first program experience 201includes a first program time axis 225 and the delayed first programexperience 203 includes a delayed program time axis 227. Thus, twopeople watching the same program at the same point on the real-time axis221 are viewing the experiential data stream from different points onthe program time axis. A corresponding playback position 229 is includedin FIG. 2 to indicate the corresponding position of the first playbackposition/playback vector 213 in the delayed first program experience 203and thus indicate that each shared social space is viewing a differentportion of the experiential data stream. Both FIG. 1 and FIG. 2 have noas-yet-unviewed data recorded.

FIG. 3 illustrates a third prior art social space 300. Again, we show afirst program experience 301 and a second program experience 303. Eachprogram experience being substantially the same and either completelyrecorded (not shown) or partially recorded using the previouslydescribed symbology. The first program experience 301 has a firstprogram start point 305 and the second program experience 303 has asecond program start point 307 that is substantially the same as thefirst program start point 305. FIG. 3 shows the experiential data streamthat provides the subject matter that can be independently navigated bythe first audience member 309 through the first program experience 301(represented by a first playback position/playback vector 313) and beingindependently navigated by the second audience member 311 through thesecond program experience 303 (represented by a second playbackposition/playback vector 315). No social space is created when the twoviewers independently navigate through the first program experience 301and the second program experience 303 such as by two people separatelywatching a DVD or recorded program on a DVR. FIG. 3 shows that the firstprogram experience 301 is not completely recorded, but has recorded someas-yet-unviewed data (as does the second program experience 303).

While the two audience members are not cooperating or communicatingabout the program experience, no social space is developed. Noticehowever, that the two audience members can cooperate to create the thirdprior art social space 300 through communication using a communicationmeans 317 such as for example, the telephone system, wirelesscommunication, or computer-assisted communication such as an e-mailsystem, instant message system, or other full- or half-duplex computerassisted communication mechanism. Thus, one audience member cancoordinate navigation with the other audience member by, for example,communicating that they are pausing the program with the hope that theother audience member will also pause. The audience members can alsoalign their playback by using countdowns, recognition of when particularscene breaks happen in the program et cetera. These methods are allclumsy. FIG. 3 also illustrates a real-time axis 321 and a first programtime axis 325 and a second program time axis 327. For the third priorart social space 300 to be established and coherent, the first playbackposition/playback vector 313 and the second playback position/playbackvector 315 need to be in approximately the same position in therespective program experiences, be on essentially the same position onthe real-time axis 321 and have a means for the audience members tocommunicate substantially as if the audience members were all local.

Some of the background art includes interactive television, instantmessaging systems, remote video conferencing, digital video, digitalaudio, network and computer technologies.

In addition, U.S. Pat. No. 6,519,771, System for Interactive ChatWithout a Keyboard, by S. E. Zenith, filed Dec. 14, 1999 disclosestechniques for combining a television signal and a networked chat roomcapability such that multiple people can communicate over the network atthe same time that a television transmission is being received andpartially displayed.

United States Patent Application Publication 2003/0101450 A1, TelevisionChat Rooms, by Marcus Davidsson et al., filed Jun. 17, 2002 alsodiscloses techniques for combining a television signal and a networkedchat room capability such that multiple people can communicate over thenetwork at the same time that a television transmission is beingreceived and partially displayed.

United States Patent Application Publication 2003/0233650 A1, VisualGroup Interface for Group Connectivity, by Melora Zaner et al., filedJun. 18, 2002 discloses a technique for creating and participating inad-hoc groups using a network. Zaner also discloses a group audio playactivity that allows group members to listen to audio in a substantiallysynchronous manner on each group member's computer. Zaner also disclosesthat every user has full control of the other group members' listeningexperience.

The capture of commentary audio clips by users or viewers of the hereindisclosed technology, and the playback of these commentary audio clipsduring asynchronous viewings supports social interaction between remoteviewers, but also creates open “loops of interaction.” The remoteviewers will be motivated to close open loops of interaction throughsynchronous conversations. The technology described and claimed by thepatent applications listed in the “Cross-Reference to RelatedApplications” and that were filed on Oct. 29, 2004 teach a heavy-weightprocess to support loop-closing interactions in the variety of occasionswhen people make contact and engage in casual conversation.

None of this art teaches or suggests, separately or combined,recordation of an information stream selected from portions of anexperiential data stream that are associated with commentary audio clipsor other commentary clips where the associated clips can be includedwith the recorded portions of the experiential data stream and presentedusing contemporary presentation devices or a limited social televisionsystem. None of this art teaches or suggests a limited social televisionsystem that provides a light-weight process for loop-closinginteractions.

It would be advantageous to provide a lightweight technology to packagesocially-enhanced viewing experiences for convenient sharing to close anopen loop of interaction.

DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a representation of a prior art social space;

FIG. 2 illustrates a representation of separate prior art social spaces;

FIG. 3 illustrates a representation of another prior art social space;

FIG. 4 illustrates one embodiment of a social television system;

FIG. 5 illustrates the environment of a social television system inaccordance with one embodiment;

FIG. 6 illustrates a high-level view of the operation of one embodiment;

FIG. 7 illustrates a thread initiation process that can be used in oneembodiment;

FIG. 8 illustrates an initialization process that can be used in anembodiment;

FIG. 9 illustrates a state exchange-output thread in accordance with oneembodiment;

FIG. 10 illustrates a display shared social space status thread that canbe used in one embodiment;

FIG. 11 illustrates a state exchange-input thread that can be used in anembodiment;

FIG. 12 illustrates a first startup phase thread that can be used in anembodiment;

FIG. 13 illustrates a second startup phase thread that can be used in anembodiment;

FIG. 14 illustrates an internal control thread process that can be usedin an embodiment;

FIG. 15 illustrates a partial representation of state informationmaintained in one embodiment;

FIG. 16 illustrates a synchronize with remote audience member process;

FIG. 17 illustrates a control panel that can be used to activate one ormore features enabled by an embodiment;

FIG. 18 illustrates a commentary audio clip presentation process;

FIG. 19 illustrates a process for associating audience member utteranceswith the experiential data stream;

FIG. 20 illustrates a process for making commentary audio clip and/orreplacement/additional content available;

FIG. 21 illustrates a first process that can be used to create contentand/or commentary extractions from a social television system; and

FIG. 22 illustrates a second process that can be used to create contentand/or commentary extractions from a social television system.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

One aspect of the technology disclosed herein is a system (such as anextension to a social television system) that extracts and storescontent and/or commentary from an experiential data stream as apresentation for later playback. The presentation can be given toanother to share the experience of viewing the experiential data streamwith the creator of the presentation. The presentation can be read andprocessed by any standard playback device (such as a DVD player, a mp3player, a tape player etc.). The presentation is predefined andconstrained to enable it to be used by commonly used playback devices.In the context of a social television system the disclosed technologyprovides a way for people to interact naturally while watching a TVprogram and to share their reactions asynchronously, and later obtainingresponses in context from people watching at different times.

One aspect of the disclosed technology determines at least one contentextent within an experiential data stream respective to at least onecommentary clip, extracts at least one portion of said experiential datastream responsive to the at least one content extent, and stores the atleast one portion of said experiential data stream.

The foregoing and many other aspects of the disclosed technology will nodoubt become obvious to those of ordinary skill in the art after havingread the following detailed description of the embodiments that areillustrated in the various drawing figures.

FIG. 4 illustrates a social television system 400 that includes anetwork 401. A first video playback device 403 and at least a secondvideo playback device 405 are able to communicate using the network 401.The social television system 400 allows audience members to experience ashared social space even if some of the audience members are remote fromother of the audience members. Thus, the first video playback device 403and the second video playback device 405 are remotely located. Forexample, these devices can be located in different localities such asdifferent homes, different rooms, different time zones et cetera. Thesedevices communicate to each other through a control link that providesdirect or indirect information transfer between the devices. The controllinks are not numbered in FIG. 4, but are indicated by arrows. Oneskilled in the art will understand that the control links can beimplemented in many ways depending on the architecture chosen for theimplementation.

The social television system 400 can scale to allow an N^(th) videoplayback device 407. Some of the video playback devices can include astorage component 409 such as a hard disk, other rotating media, ormemory that can be used to store an experiential data stream (amulti-media program that can include a video component and/or an audiocomponent and/or potentially other components) such as televisionprogramming, home video, digitized movie film, audio, still pictures etcetera (including digital video recorders, DVDs, CDs et cetera). Thevideo playback device can also include a video component 411 thatprepares the visual aspect (if any) of the experiential data stream forpresentation to one or more local audience members through a videodisplay component 413. Some embodiments of the first video playbackdevice 403 can include a DVD component 415 and/or a VCR component 417(as in the second video playback device 405). The social televisionsystem 400 can also include an optional content provider service 419that provides an experiential data stream that, when presented to anaudience member, provides an experience to the audience member. Theoptional content provider service 419 can provide the experiential datastream over a broadcast channel, over a cable channel, over a satellitechannel, over the Internet or other LAN or WAN. The experiential datastream can be recorded on the storage component 409 as the experientialdata stream can be received and stored as a recorded video/audio program420 for later playback.

A networked computing device 421 that can be directly connected to thenetwork 401 or indirectly connected to the network 401 (for example,through a router (not shown) or the first video playback device 403, etcetera) also can be used to supplement communications between the remoteaudience members and the local audience members. This capability canalso be included in the first video playback device 403 as well as theothers.

The experiential data stream can also be provided from a program DVD 423that can be accessed using the DVD component 415, a VCR tape accessedusing the VCR component 417, or from a computer, et cetera as well asfrom the recorded video/audio program 420 stored on the storagecomponent 409 of a video playback device. Other computer readable datacarriers can also be accessed to obtain the experiential data stream.One important aspect of the first video playback device 403 is that ithas access to a completely or partially recorded version of theexperiential data stream.

An audio input component 427 on the video playback device (orcorresponding functionality on some other device such as a telephone orthe networked computing device 421) can be used to capture transitoryaudio from the local audience members using a microphone (not shown).The captured transitory audio (as well as program audio, transitoryaudio or commentary audio clips received from the remote audiencemembers and other commentary audio clips) can be reproduced using anaudio output component 429 through the video display component 413 orseparate audio component (not shown).

In addition, the captured audio (or other audience member communicationsuch as instant messaging and UNIX® talk) can be processed by aconversation finder (described in the previouslyincorporated-by-reference published United States patent applications)to allow, for example, side conversations to occur between remote andlocal audience members. The conversation finder can reside on thenetworked computing device 421, on the local device, or distributedamong some selection of devices. An example of the conversation finderis the floor analysis module 303 of FIG. 3 of theincorporated-by-reference published United States patent applications.

The microphone can be used to capture transitory audio from localaudience members who are experiencing the experiential data stream. Thetransitory audio can also be transmitted to a remote audience member'sremote device for immediate presentation to the remote audience membersas part of the shared social space. The transitory audio can also berecorded as a commentary audio clip for presentation responsive to theplayback position of the experiential data stream (and a copy of thecommentary audio clip can also be transmitted to the remote device). Inorder to capture transitory audio alone without corruption fromsimultaneously presented audio (for example, audio from the experientialdata stream, a commentary audio clip, transitory audio from a remoteaudience member or a mixture of these) the raw signals from themicrophone can be processed by echo cancellation algorithms or othersignal processing techniques as is known by one skilled in the art.These algorithms may be given as input not only the microphone signalsbut also the signals of the audio presented by the local device as wellas characterizations of the room and audio system such as might beobtained by playing, recording, and analyzing test tones. One advantageof combining the handling of the experiential data stream, the recordingof transitory audio, and the presentation of commentary audio clip inone system is the ability to perform this signal processing based on allrelevant signals. The microphone may be replaced by a set of microphonesin such arrangement and with such signal processing as may provide forthe highest quality recording of transitory audio.

The control link used in the shared social space can be supported by apeer-to-peer or multi-cast architecture. In addition the control linkcan also be supported by a server architecture where communicationsbetween the video playback devices is accomplished through a sharedsocial space service device 431 that can also be configured to maintainthe state of each video playback device and to redistribute the state tothe video playback devices. Thus, one embodiment allows the transmissionof state from the server to all the devices.

The social television system 400 can also include an optional programinsertion provider service 433 that allows replacement/additionalcontent provided by the optional program insertion provider service 433to be presented instead of (or in addition to) specific portions of theexperiential data stream. The inventors expect that this content can beadvertising content directed to the members of the shared social spaceas well as still pictures, replacement video segments and music. Theinventors also expect that the replacement/additional content can bepresented instead of targeted portions of the experiential data streamfor the entire shared social space or for selected local audiencemembers where the replacement/additional content of the targetedportions of the experiential data stream are different versions of theexperiential data stream that conform to program rating requirements (ordesires), parental controls or requirements of the V-chip. Further thereplacement/additional content can be presented instead of commercialsin the experiential data stream and/or presented to supplement thecommercials in the experiential data stream. In some embodiments thereplacement/additional content can be stored separately from theexperiential data stream such that a subsequent presentation of theexperiential data stream does not include the replacement/additionalcontent. Other embodiments actually replace a portion of theexperiential data stream or insert the replacement/additional contentinto the recorded experiential data stream. The replacement/additionalcontent can also include (separately or combined with the video)commentary audio clips.

Furthermore, video or stills from a video camera 435 can be used toprovide additional control and state information as well as enhancingthe shared social space by allowing the remote audience members to seethe local audience member(s). This picture information can be displayedwithin a Picture-In-Picture format or on other devices that are notshown. In addition, such video or stills can be used to determine when alocal audience member enters or leaves the local viewing area (and insome embodiments, used to identify the local audience member).

The first video playback device 403, as well as the others, can alsoinclude a memory 437 that can be accessed by a CPU (not shown) and thatcan contain a computer program 439 to instruct the CPU to perform theprocesses described herein as well as other processes. The computerprogram 439 can be embodied in a computer-usable data carrier such as aROM within the device, within replaceable ROM, in a computer-usable datacarrier such as a memory stick, CD, floppy, DVD or any other tangiblemedia. In addition the computer program 439 can be provided from devicesaccessed using the network 401. One skilled in the art will understandthat the network 401 is another computer-usable data carrier. Oneskilled in the art will also understand that the computer program 439and the experiential data stream are completely different types of data.The computer program 439 causes a computer or other data processor toperform particular processes while the experiential data stream containsdata that is presented to an audience member.

A control device 441 can be used to allow operator commands to be sentto any of the playback devices. Such a device can be a control panel onthe playback device itself, a wired or wireless web interface, aninfrared device, et cetera. Some of the inventive aspects of theembodiments are controlled by the audience member through the use of thecontrol device 441. Some aspects of the control device 441 aresubsequently described with respect to FIG. 17.

The inventors use the term “synchronization” to refer to the processused to bring the local audience member's playback position intoalignment with a remote audience member's playback position. Theinventors use the term “tracking” to refer to the process of maintainingthe local audience member's playback position in alignment with a remoteaudience member's playback position. For example, activation of the“follow” user selectable control generally causes the local audiencemember's device to first synchronize with the followed device (hence therespective playback positions are aligned), and then having the localdevice monitor the difference between the playback positions and adjustthe local device to track the followed remote device. Playback positionsare “aligned” or in “alignment” when the difference between the playbackpositions is sufficiently small that the audience members who areexperiencing the experiential data stream at the separate playbackpositions experience substantially the same content.

Audio from the remote audience members can be presented to the localaudience member. If the audio from the local audience member or remoteaudience member is stored and indexed to the experiential data stream(such as with an audio trigger position) that audio is considered to becommentary audio clip in that commentary audio clip can be reproducedduring an experience of the experiential data stream. Transitory audio(audio that is not a commentary audio clip—thus not, or not yet, indexedto the experiential data stream) can also be presented to theappropriate audience members near the real time the transitory audio isuttered, but will not be stored for presentation in subsequentexperiences of the experiential data stream. The commentary audio cliphas an audio duration that is the time required to utter the commentaryaudio clip (the audio duration can be adjusted by changing the playbackspeed for the commentary audio clip with or without pitch compensation).

The local audience member can separately enable or disable presentationof each commentary audio clip, enable or disable presentation of groupsof commentary audio clips, record their own commentary audio clips, andcan combine selected commentary audio clips to make a group ofcommentary audio clips (possibly subject to copyright protectionmechanisms).

The commentary audio clips can be grouped and provided individually orin groupings. Thus, a local audience member can obtain a commentaryaudio clip group that is a series of commentary audio clips made by, forexample, a single expert on the content presented in the experientialdata stream. Other commentary audio clips can be grouped by remoteaudience member's device identifications and/or date and time. Multiplecommentary audio clip groups can be enabled to be presented during thepresentation of the experiential data stream.

The commentary audio clips can contain content that is analysis audio,opinion audio, critical audio, parody audio, entertainment audio orparticipant audio related to said experiential data stream. Thus, anaudience member can obtain and/or purchase commentary audio clips thatprovide audio analysis and/or opinion and/or critique by an expert onthe experiential data stream, that provide parodies the experientialdata stream, that provides an audio enhancement for the experientialdata stream to further entertain the audience member and/or that providecommentary by a person who participated in the creation of theexperiential data stream.

FIG. 5 illustrates a social television system experience 500 that isenabled by the social television system 400 without the problemsinherent in the prior art systems of FIG. 1 through FIG. 3. Thisrepresentation of the social television system experience 500 includes areal time axis 501, a first recorded program 503, a second recordedprogram 505 and a third recorded program 507. Each of these programs canbe completely or partially recorded (for example, if the audiencemembers have recorded the same satellite television program fromdifferent feeds). These recorded programs can also reside on DVDs, or bea mixture of DVD and recorded feed or other heterogeneous mixture ofexperiential data stream formats.

The recorded programs have a program time axis 509. For discussionpurposes, recorded video programs are assumed to be in MPEG-2 formathaving a group-of-pictures (GOP) organization of I-frames, B-frames andP-frames. Other modern video recording formats have similarcharacteristics such that one skilled in the art can apply the teachingsherein to those formats.

In this social television system experience 500, a first audience member511 is navigating through the first recorded program 503 at a firstplayback position/playback vector 513; while a second audience member515 is navigating through the second recorded program 505 at a secondplayback position/playback vector 517; and while a third audience member519 is navigating through the third recorded program 507 using a thirdplayback position/playback vector 521. Each audience member's playbackposition is shown at a different point on the program time axis 509 butat the same point on the real time axis 501.

Using embodiments of the subsequently described inventive technology,the first audience member 511, the second audience member 515, and thethird audience member 519 form a shared social space 523 even if some ofthe audience members are located remotely from the others. One aspect ofthe shared social space is that, in general, the audience members'playback positions and-playback vectors are often substantially thesame.

Some of the difficulties in establishing the shared social space 523 arethat the actual start positions (a first program start point 525, asecond program start point 527, and a third program start point 529) maybe different for each of the recorded programs. Thus, some embodimentsare disclosed that locate a common reference point for the start of therecorded program. Another difficulty occurs when the playback positionsof the audience members are different such that one or more audiencemembers are out of alignment with other of the audience members (thus,reducing the coherency of the shared social space). Another difficultyis that the playback vector may be different for some of the audiencemembers such that the respective playback positions move out ofalignment. Yet another difficulty is that existing DVR and DVD devicecontrols do not provide for the ability to align the audience memberswithin the shared social space 523. Still another difficulty is that theaudience members need to be able to communicate about the program withinthe shared social space 523. This communication in the shared socialspace 523 needs to be compatible with traditional interpersonalcommunication by audience members in, for example, the firstrepresentation of a prior art social space 100 of FIG. 1.

The playback vectors in FIG. 5 are shown with different directions andlengths to indicate that these vectors can be adjusted through audiencemember command or automatically to keep the shared social space 523coherent. Some of these techniques are subsequently described withrespect to at least FIG. 16.

One advantage of the shared social space 523 is that the separateaudience members can interact with each other in a manner as if theywere in the same physical location. In one embodiment, one audiencemember (a master audience member) can be designated to operate a controldevice. In this instance, as all the other audience members are slavedto the master, the master's operation of the control device will alsocontrol the viewing of the experiential data stream by the otheraudience members. In another embodiment, each audience member can“follow” a specific audience member, or follow a virtual audiencemember. The virtual audience member can be the mean/median playbackposition calculated from all or some selection of the audience members.Some embodiments also allow an alignment control that allows oneaudience member to reach alignment with and synchronize playback withanother audience member either as a jump synchronization, or over aperiod of time to allow the local audience member to maintain contextwhile reaching alignment through use of, for example, acontext-displayed synchronization or a social-context synchronization.Other embodiments allow each audience member to use their control toaffect all the other audience members in the shared social space (forexample, any audience member can pause all audience members' devices).

One skilled in the art will understand that each audience member needsto have a prerecorded (at least partially) version of the experientialdata stream (the program) that is substantially the same as thatrecorded by the other audience members. The experiential data stream canbe an audiovisual program completely, partially, or a mixture ofcompletely or partially (for example, completely recorded in one timezone while being recorded in another time zone), recorded on a DVR, aprogram recorded on a DVD, a recorded audio file in any format that canbe stored et cetera

It is important to realize, that none of the audience members' devicesare supplying the experiential data stream to other audience members'devices. Each device can be providing both playback of the experientialdata stream to the associated audience member and at least informationacross the control link of its internal state to the other devices. Thecontrol link can also be used to transfer audio and/or pictureinformation between the audience members.

In addition, the audio received from a remote audience member can berecorded and indexed to the remote audience member's playback positionat the time the audio was uttered when the audio is stored on the localaudience member's device. Thus, commentary audio clips uttered by aremote audience member having a playback position ahead of the localaudience member's playback position, can be stored and indexed to theremote audience member's playback position (the audio trigger position).The commentary audio clip can be presented to the local audience memberwhen the local audience member reaches the audio trigger position in theexperiential data stream. In addition, commentary audio clips uttered bya remote audience member can also be indexed to the experiential datastream irrespective of the playback position of the local audiencemember and irrespective of whether the experiential data stream has yetbeen recorded by the local audience member's device at the audio triggerposition.

FIG. 6 illustrates a high level shared social space process 600 thatshows an overview of the procedures used by one embodiment. The highlevel shared social space process 600 initiates at a start terminal 601responsive to a command from a prospective audience member. A ‘joinguests’ procedure 603 allows the prospective audience member to be addedto the shared social space. Many techniques exist that can be used toaccomplish this. These techniques include using a web-based enrollment,techniques similar to instant message (IM) buddy chat, techniquessimilar to selecting players in a multi-player computer or video game,et cetera. Furthermore, the ‘join guests’ procedure 603 can be automatedfor the audience members such that the expected experiential data streamis automatically recorded at each audience member's device, and theshared social space established at an agreed-upon time.

An ‘establish human communication’ procedure 605 can establish a humancommunication channel for human communication between the joinedaudience members through traditional means or by use of the controllink.

At some point, an agreement is reached by the audience members who havejoined the shared social space as to what experiential data stream theydesire to experience. This agreement can be accomplished by a ‘selectexperiential data stream’ procedure 607.

The ‘join guests’ procedure 603 and the ‘establish human communication’procedure 605 can be accomplished through heterogeneous communicationmechanisms as well as homogeneous communications. The humancommunication channel can be established using, for example, but withoutlimitation, VoIP, POTS, the control link, instant message, e-mail, atext-based communication capability, web page enrollment capability, andcomputer-assisted audio. One skilled in the art will understand thatthere are many ways to select the experiential data stream for use inthe shared social space (for example, this functionality can be includedin the ‘join guests’ procedure 603). Information received over the humancommunication channel can include text, video, still pictures, audiodata, and any data that can be converted to audio form, text form, videoform, or picture form. For example, textual information can be convertedinto computer generated audio that represents the text and that audiocan be immediately presented (as transitory audio) or presented andstored (capable of being presented) as a commentary audio clip.

Once the audience members are joined and the experiential data streamselected for the shared social space, a ‘start threads’ procedure 609invokes the process illustrated in FIG. 7 to initiate the threads thatprovide the inventive services in this embodiment.

Once the threads are started, the high level shared social space process600 continues to a ‘completion’ decision procedure 611 that detects whenthe local audience member wishes to leave the shared social space. Whilethe local audience member remains in the shared social space, the highlevel shared social space process 600 periodically loops back to the‘completion’ decision procedure 611. When the local audience membercommands an exit from the shared social space (or if an exit conditionis otherwise detected), the high level shared social space process 600continues to a ‘cleanup’ procedure 613 that terminates executingthreads, releases resources used by the high level shared social spaceprocess 600 as well as any other thread associated with establishmentand maintenance of the shared social space 523. Finally, the high levelshared social space process 600 completes through an end terminal 615.

FIG. 7 illustrates a thread initiation process 700 that can be invokedby the ‘start threads’ procedure 609 of FIG. 6. The thread initiationprocess 700 initiates at a start terminal 701 and continues to an‘establish device communication’ procedure 703 that establishescommunication over the control link (using for example but withoutlimitation, UDP or TCP protocols) between the relevant devicescomprising the shared social space and/or supporting the shared socialspace (for example, the shared social space service device 431).

Once communication is established between the relevant devices, thethread initiation process 700 continues to an ‘initialization’ procedure705 that initiates the input and output state exchange threads and thatis subsequently described with respect to FIG. 8.

A ‘start control thread’ procedure 707 initiates the control thread thatis subsequently described with respect to FIG. 14.

In addition, a ‘start status display thread’ procedure 709 initiates thedisplay thread that is subsequently described with respect to FIG. 10.

Once the input and output state exchange, control, and status displaythreads (as well as any other ancillary service threads) areestablished, an ‘enable extended commands/functions’ procedure 711enables extended commands and functions such as the shared social spacethat enable interaction between the audience members and between theirrelevant devices in the shared social space as well as extended commandsand functions for processing commentary audio clips and other sharedsocial space and stand-alone features disclosed herein.

Once the service threads are established and the shared social spacefunctions enabled, the thread initiation process 700 terminates via anend terminal 713.

One skilled in the art will understand that the relevant devices aredetermined depending on the type of communication architecture used tosupport the shared social space. In a peer-to-peer architecture, therelevant devices would generally consist of each audience member'sdevice that can access the experiential data stream. In a client serverarchitecture, the relevant device could simply be an identified computerthat accumulates and distributes information from each audience member'sdevice to the other audience members' devices.

FIG. 8 illustrates an initialization process 800 that can be invoked bythe ‘initialization’ procedure 705 of FIG. 7, that initiates at a startterminal 801 and continues to a ‘start threads for phase 1 and 2startup’ procedure 803. The ‘start threads for phase 1 and 2 startup’procedure 803 starts the threads used to determine a starting point inthe experiential data stream recorded in each of the audience members'devices. The ‘start threads for phase 1 and 2 startup’ procedure 803 issubsequently described with respect to FIG. 12 and FIG. 13.

Then the initialization process 800 continues to a ‘start stateexchange-output thread’ procedure 805 that initiates a thread that issubsequently described with respect to FIG. 9.

In addition, a ‘start state exchange-input thread’ procedure 807initiates a thread that is subsequently described with respect to FIG.11. Once the state exchange threads are initiated the initializationprocess 800 completes via an end terminal 809.

Some embodiments can include an audio input thread (not shown) and/or avideo input thread. These threads can be used to capture localtransitory audio or video for transmission to the remote audiencemembers (and, in some embodiments, the captured transitory audio orvideo can be processed by a conversation finder or an image recognitionsystem).

FIG. 9 illustrates a state exchange-output thread 900 that can beinvoked by the ‘start state exchange-output thread’ procedure 805 andthat initiates at a start terminal 901. Once initiated, a ‘detect statechange’ procedure 903 detects a state change in the local audiencemember's device. Once a state change is detected, a ‘send local state’procedure 905 sends the changed state of the local audience member'sdevice to the other audience members' devices (in accordance with thearchitecture being used). The state exchange-output thread 900 thencontinues back to the ‘detect state change’ procedure 903 to detect thenext state change. One component of the information that can be sent bythe state exchange-output thread 900 (when the experiential data streamis in the MPEG-2 format) is the identification of the next Group ofPictures (GOP) that will be displayed by the local audience member'sdevice.

One skilled in the art will understand that another embodiment may usesynchronous transmission of the device's state such that the state canbe periodically sent (changed or not).

FIG. 10 illustrates a display thread process 1000 that can be invoked bythe ‘start status display thread’ procedure 709, that initiates at thestart terminal 1001 and continues to a display status 1003 that detectsa change of state for one or more of the audience members' devices anddisplays the new or changed status; or causes the new or changed statusto be displayed. Once the status is displayed, the display threadprocess 1000 loops back to the display status 1003. The status can be ofthe local audience member's device, of the shared social space, or anyother status of interest to a local audience member.

FIG. 11 illustrates a state exchange-input thread 1100 that can beinvoked by the ‘start state exchange-input thread’ procedure 807,initiates at a start terminal 1101 and continues to a ‘receive GOPalignment ID and state’ procedure 1103. The ‘receive GOP alignment IDand state’ procedure 1103 receives state information (or change of stateinformation) as well as a group of pictures (GOP) alignmentidentification (in an embodiment that has a group of pictures, forexample MPEG-2) from the audience member's device that indicates theaudience member's playback position in the experiential data stream. Onreceiving the GOP alignment ID, an ‘update state’ procedure 1105 updatesthe state of the identified device. A ‘tracking’ decision procedure 1107determines whether the device that received the GOP alignment ID istracking the playback position of a remote device.

If the device that received the GOP alignment ID is tracking or is inone of the “syncing” states, the state exchange-input thread 1100continues to a ‘compare playback position with leader’ procedure 1109that compares the local audience member's playback position in theexperiential data stream with the remote audience member's playbackposition in the experiential data stream.

In some embodiments that support heterogeneous formats for theexperiential data stream, the process for tracking can be refinedaccordingly. For example, if the followed audience member fast-forwardspast a commercial break that does not exist in the format of the localaudience member's experiential data stream, the local audience member'sdevice pauses or slows while the followed audience member fast-forwardsinstead of attempting to skip content that does not exist in the formatof the experiential data stream experienced by the local audiencemember.

In some embodiments an ‘adjust playback vector’ procedure 1111 adjuststhe playback vector of the experiential data stream for the localaudience member so that the local audience member's viewing experiencewill align with the playback position of the remote audience member overa period of time (for example, as in a context-displayed synchronizationor a social-context synchronization—subsequently described with respectto FIG. 16). Thus, the local audience member is exposed to the contextof the intervening portion of the experiential data stream whilesynchronizing with the remote audience member. The audio playback canalso be adjusted to keep the transitory audio and/or commentary audioclips in synchronization with the experiential data stream by any of anumber of techniques well known in the art (for example, pitchcompensation). In other embodiments, the ‘adjust playback vector’procedure 1111 simply “jumps” to the remote audience member's playbackposition. Yet other embodiments provide both mechanisms. Additionaldetails related to the process of aligning the local audience member'splayback position with a remote audience member's playback position aresubsequently provided with respect to FIG. 16. One skilled in the artwould understand how to implement and coordinate the ‘adjust playbackvector’ procedure 1111 with the process described with respect to FIG.16.

Some embodiments require that the viewing distance between playbackpositions be larger than a threshold before adjusting the playbackvector. This reduces speed oscillations in the playback. One skilled inthe art will understand that the playback vector of the remote audiencemember's device will need to be determined and incorporated into thiscomputation if that device's playback vector is other than nominal. Fromthis disclosure as a whole and from the above, such a one will alsounderstand that the device also monitors its own playback position.

The state exchange-input thread 1100 then returns back to the ‘receiveGOP alignment ID and state’ procedure 1103. The state exchange-inputthread 1100 and the synchronize with remote audience member process 1600(subsequently described with respect to FIG. 16) can cooperate toestablish and maintain alignment of the playback positions of the localaudience member and the followed remote audience member (by tracking theremote audience member's playback position). However, one skilled in theart will understand that some embodiments allow a local audience memberto align with a remote audience member without requiring that the twoaudience members remain in alignment.

If the ‘tracking’ decision procedure 1107 determines that the localaudience member is not in the “following” state, the stateexchange-input thread 1100 continues to a ‘determine social/viewbalance’ procedure 1113 that evaluates the social versus viewing balanceand determines whether and how to adjust the playback of theexperiential data stream accordingly.

From experiments performed by the inventors, it is clear that audiencemembers are very good at knowing when they can start talking betweenthemselves in response to lulls in the experiential data stream (such asduring commercials, lulls in dialog et cetera). The difficulty is thatonce the audience members start a conversation, they find it much harderto stop the conversation when the experiential data stream again becomesinteresting. Thus, the social conversation between the audience memberscan overlap with the resumption of interesting program content.

The ‘determine social/view balance’ procedure 1113 can monitor theutterances vocalized by the audience members (either just the localaudience member or all the audience members in the shared social space)and can also monitor characteristics of the experiential data stream.For example, as the end of a commercial break approaches and if theaudience members' audio streams are sufficiently active, the ‘determinesocial/view balance’ procedure 1113 can condition the ‘adjust playbackvector’ procedure 1111 to slow or pause, the presentation of theexperiential data stream; or present other material instead of theexperiential data stream until the audience members have finished theirconversation and are ready to resume experiencing the experiential datastream. Thus, the ‘determine social/view balance’ procedure 1113 candetermine when the audience members have completed their conversationand can resume nominal playback accordingly by conditioning the ‘adjustplayback vector’ procedure 1111. Thus, the first video playback device403 can adjust to become less disruptive to the audience members withinthe shared social space.

In addition, the ‘determine social/view balance’ procedure 1113 canoptionally determine whether the audience members' conversations havereached a level where the audience members are substantially engaged insocial conversation and not experiencing the experiential data stream aspresented (thus, the audience members' conversations should havepriority over the presentation of the experiential data stream). In thissituation, some embodiments can disable the presentation of theexperiential data stream and can present replacement/additional contentsuch as an advertisement provided by the optional program insertionprovider service 433 instead of the content from the experiential datastream until the audience members' conversations complete or slacken(thus providing a non-scheduled commercial break).

An audience member's device that is tracking can still be subject to theoperation of the ‘determine social/view balance’ procedure 1113. Forexample, the tracked device has access to the amount of conversations inthe shared social space. If the tracked device performs as above, thetracking devices can follow the tracked device accordingly.

A conversation finder can be used in some embodiments to help determinethe social/view balance by, for example, monitoring the local and remoteutterances to detect active conversations between the audience members.

In some embodiments, once the start of a commercial break is detected,and if each of the audience members' devices have recorded data of theexperiential data stream that extends past the commercial break, eachaudience member's device can simply fast forward or skip to the portionof the experiential data stream that resumes after the commercialbreak—thus skipping past the commercial break.

As just mentioned, during the period when the audience members areconversing, other material such as replacement/additional content can bepresented instead of the content from the experiential data stream. Theconcept of replacement/additional content can be expanded to useremotely or locally stored still pictures or slide shows, locally storedvideo, additional commercials provided by the optional program insertionprovider service 433, images of some number of audience members, etcetera

Once the ‘determine social/view balance’ procedure 1113 determines thatthe audience members have completed their conversation, it canimmediately enable the presentation of the experiential data stream thusresuming the experiential data stream, or it can wait until thecurrently presented material completes (thus allowing for thereplacement/additional content, for example an inserted advertisement,to complete prior to enabling the resumption of the presentation of theexperiential data stream). Because each remote audience member's devicemay have different length commercial breaks from that of the localaudience member's device, the ‘determine social/view balance’ procedure1113 can also determine when all of the audience members' devices havecompleted their commercial break period.

The replacement/additional content can be provided from the optionalprogram insertion provider service 433 or from storage on the localaudience member's device or network. The replacement/additional contentcan be, for example, but without limitation, a still picture, a statusdisplay of the shared social space, a text and/or graphicalrepresentation of the shared social space, a slide show of stillpictures, additional advertising, images of audience members, or othercontent related to or unrelated to the experiential data stream.Furthermore, when the ‘determine social/view balance’ procedure 1113detects a commercial break in the experiential data stream, it cancondition the ‘adjust playback vector’ procedure 1111 to partially orcompletely replace the content of the commercial break withreplacement/additional content (including extending the commercial breakto provide additional advertising). Thus, the presented advertisementscan be targeted to individual audience members and/or to the sharedsocial space.

Furthermore, replacement/additional content can be presented instead ofa portion of the experiential data stream content to accommodatesupplemental content, parody content, comedy content, content responsiveto parental controls, V-chip requirements, or program ratingrequirements or desires. The replacement content can be selected basedon any selection criteria such as, or similar to the above, or impliedherein. In some embodiments the selection criteria can be specified orinfluenced by the audience member (for example with rating preferences).In some embodiments the selection criteria can be specified orinfluenced by the provider of the replacement/additional content (forexample, by a provider of advertising content). Some embodiments canprovide support for both of these. In some embodiments, the selectioncriteria can be provided to the optional program insertion providerservice 433 as a content parameter in the request for content.

In addition, the replacement/additional content can also be used topresent “director's cut” additional content in addition to the contentin the recorded experiential data stream as well as providing alternateendings. Furthermore in a shared social space where there is aheterogeneous experiential data stream environment (for example, wherethe local audience member is using a DVD while a remote audience memberis using a recorded satellite feed that includes advertisements) theremote audience member's device can detect a commercial break andprovide the local audience member's device with that state information.The local audience member's device can then present advertisementsprovided by the optional program insertion provider service 433 duringthe period that the commercial break is imposed on the remote audiencemember. Thus, for example, advertisements can be included in thepresentation of the content of a DVD even though there are noadvertisements contained in the experiential data stream recorded on theDVD.

One skilled in the art will understand that replacement/additionalcontent can be provided from the optional program insertion providerservice 433 in a push mode, a pull mode, a cache preload mode, atangible media mode, or other delivery mode. In addition, one skilled inthe art will understand how to provide records back to the optionalprogram insertion provider service 433 to account for viewings. Such aone will also understand that the same replacement/additional contentcan be displayed to all audience members in the shared social space aswell as having different replacement/additional content targeted to someor all of the audience members' devices.

The replacement/additional content can be configured to specify whichportion of the experiential data stream the replacement/additionalcontent replaces. This can be accomplished by specifying or otherwiseidentifying an insertion coordinate and a resumption coordinate in theexperiential data stream for which the replacement/additional content isto be substituted during presentation. For example, if the insertioncoordinate and the resumption coordinate are the same, thereplacement/additional content is presented at that point in theexperiential data stream and no content from the experiential datastream is lost. If the coordinates define a portion of the experientialdata stream, the replacement/additional content is presented instead ofthe content within that portion of the experiential data stream. Atcompletion of the presentation of the replacement/additional content,the presentation of the experiential data stream can continue startingat the resumption coordinate specified in the replacement/additionalcontent.

The replacement/additional content can also be configured to filterportions of the video generated by the experiential data stream. Forexample, the replacement/additional content can include maskinginformation, pixilating information, transparency information et ceterathat can be applied to the video generated by the experiential datastream to obscure, enhance, pixilate portions of a video frame generatedby the experiential data stream.

One skilled in the art will understand that there exist other techniquesthat can be used to identify a portion of the experiential data streamto be replaced such that absolute coordinates need not be specifiedwithin, or associated with the replacement/additional content. In suchembodiments, the insertion coordinate and the resumption coordinate willgenerally be determined so that the device can detect proximity of itsplayback position in the experiential data stream with the insertioncoordinate. Such a one will also understand there exist many equivalentways to differentiate between insertion content and replacement contentthat are equivalent to what is described herein.

One aspect of the technology disclosed herein is a server for thereplacement/additional content that resides on the network 401 andprovides the replacement/additional content responsive to a request forthe replacement/additional content from an audience member's device.This request can be invoked by the audience member in many ways, forexample by the audience member explicitly invoking the request, by theaudience member conditioning the audience member's device to issue therequest periodically, or in a response to satisfaction of one or moreconditions for issuing the request. Furthermore the request can besubmitted by the audience member to the server though a web or otheruser interface, by telephone, by e-mail, or by any other process used torequest information from the server. This request can be configured tocause the server to provide the replacement/additional content accordingto parameters within the request including whether thereplacement/additional content is to be provided in a push mode, a pullmode, a cache preload mode, et cetera. The request can also includecontent parameters to condition the content selection according to theaudience member's preferences. The replacement/additional content can beused to supplement the experiential data stream such as presenting thereplacement/additional content in addition to, or instead of content inthe recorded experiential data stream. The content parameters caninclude an identification of the experiential data stream, userspecified parameters that can be used to better select targetedadvertising for the audience member, experiential data stream format(for example, letterbox or normal presentation), content limitations(for example content rating limitations), et cetera

One function of a first startup phase thread 1200 and a second phasestartup thread 1300 (both invoked from the ‘start threads for phase 1and 2 startup’ procedure 803 and subsequently described with respect toFIG. 12 and FIG. 13) can be to verify that the experiential data streamavailable at the local audience member and the remote audience memberare substantially similar as well as to determine the starting pointwithin the experiential data stream for the audience members. It isinstructive to evaluate some of the possible arrangements of theexperiential data stream under consideration.

The first and simplest situation is where each audience member has thesame version of the experiential data stream (for example, where eachaudience member has a DVD of the same version of a movie). In this case,each I-Frame and GOP is identical and the start GOP can be the firstGOP. The synchronization GOP (see below) can also be the first GOP orsome future GOP.

A second situation is where each audience member is receiving a feedfrom the same high-quality channel. For example, a United States WestCoast Satellite feed of a particular program on the same channel willhave the characteristics that should be handled when setting up a sharedsocial space. In particular, we assume that a significant portion of thestart of the program or the entirety of the experiential data stream isalready recorded (such as by a DVR). Some of the issues that must beconsidered in this situation are that each audience member's DVR may nothave started recording at the same time; that the internal clocks ofeach individual DVR may not be the same; that different audience memberswill have different signal drop outs from local interference; and thatthe different audience members may have recorded the same program indifferent formats (for example, widescreen versus normal).

In this situation, we must verify that the selected experiential datastream at the local audience member's device and the one at the remoteaudience member's device are substantially similar. Then we must locatea starting GOP where not all of the initial GOPs in the experientialdata stream are available to all audience members. In addition, not allof the GOPs need be identical because of local interference (forexample, drop-outs from passing airplanes disrupting satellitecommunication, storms, interference from lightning, et cetera) that mayhave corrupted the signal recorded by the DVR. Thus, in this situation,we need to determine the start point for the experiential data streamand determine where in the experiential data stream we expect theaudience members to align.

The need for a synchronization GOP (or other synchronization point) isthe result of allowing those who have completely recorded the start ofthe experiential data stream to view that portion of the experientialdata stream while still allowing others who did not record the start ofthe experiential data stream to be able to align with those that did.Determining the starting GOP and synchronization GOP is made moredifficult because some of the recorded GOPs may be different between theaudience members.

A third situation is where some of the audience members record theexperiential data stream off of one channel, while other audiencemembers record off a different channel. For example, the sameexperiential data stream, but from an East Coast Feed versus the WestCoast Feed. In this case, the experiential data stream is sent atdifferent GMT times. Thus, some audience members may have the entireprogram recorded, while others have only a partial recording. Inaddition, the commercials may be dependent on the feed.

Yet another situation is where some of the audience members are on acable channel in various cities, some on a satellite channel, yet othersare recording the experiential data stream from a broadcast channel invarious cities. In this situation, there can be significant differencesin the recorded content between each of the audience members. Thesedifferences include different commercials, different amounts ofinterference, and different editing of the experiential data stream.

Yet another situation is where the one or more DVRs stores theexperiential data stream in a different format than other of the DVRs.Embodiments can address these conditions separately or in combination.

Yet another situation that can apply to each of the previously describedsituations is when a non-member of the shared social space desires tojoin the shared social space after the existing audience members havestarted experiencing the experiential data stream. In this case theexperiential data stream recorded on the non-member's device may nothave initial data for the complete experiential data stream.

One skilled in the art will understand the experiential data stream ateach audience member's device is generally substantially similar, andthat the playback position provided by the remote audience member'sdevice to the local audience member's device is related to the localaudience member's experiential data stream. Where the experiential datastreams are identical and without replacement/additional content, thereceived playback position is strongly related to the local audiencemember's experiential data stream. Where, for example, the localaudience member's experiential data stream is from a DVD and the remoteaudience member's experiential data stream has been recorded from abroadcast, the playback position provided by the remote audiencemember's device is related to the local audience member's experientialdata stream. In such configurations, periodic re-alignments may berequired using techniques similar, for example, to those described withrespect to FIG. 12.

I-Frames and/or GOPs (or any other audio, visual and/or video unit inthe experiential data stream—collectively referred to as a contentsample) can be characterized by numerous methods (such as preparing achecksum, a cyclic redundancy check, a digest or any other techniquethat can, with high probability, determine whether two blocks of dataare identical) from the image portion of the I-Frame to characterize theI-Frame as a function of the visual data contained in the I-Frame. Inaddition, I-Frames can be characterized by determining statisticalvalues of the GOP associated with each I-Frame to determine if theI-Frames are substantially similar. Furthermore, I-Frames can becharacterized by the time-stamps within the I-Frame. One skilled in theart will understand that characterizations similar to these can beapplied to data streams that are not in the MPEG-2 format. The audioinformation in the content sample can also be characterized.

In addition, GOPs can be characterized by numerous methods to detectsubstantially similar GOPs. These methods, known to one skilled in theart include feature extraction techniques for color, texture, shape,motion and audio features and can generate a multidimensional vectorthat characterizes the GOP. These characterizations can also be used todetermine the most similar GOP between in each audience members'experiential data stream.

FIG. 12 illustrates the first startup phase thread 1200 that can beinvoked by the ‘start threads for phase 1 and 2 startup’ procedure 803of FIG. 8, initiates at a start terminal 1201, and continues to a‘receive characterization information’ procedure 1203. The ‘receivecharacterization information’ procedure 1203 receives characterizationinformation from the control link that was sent by a ‘sendcharacterization information’ procedure 1309 that runs in the remotedevices and that is subsequently described with respect to FIG. 13. Oneskilled in the art will understand that the ‘receive characterizationinformation’ procedure 1203 can be implemented as a thread and thatsubsequent condition flags (not shown) can be used to indicate whencharacterization information is received from all or a selection ofaudience members' remote devices. A ‘wait for local characterizationcompletion’ procedure 1205 determines when the I-Frames/GOPs (or othercontent samples) on the local device have been characterized. After thelocal I-Frames/GOPs are characterized, and characterizations of contentsamples from all or a sufficient number of remote devices have beenreceived over the control link, the first startup phase thread 1200continues to an ‘align I-frames’ procedure 1207 that uses the contentsample characterization information to verify that each of the audiencemembers' devices have access to the same (or substantially similar)experiential data stream, to determine characteristics of each remotelyrecorded experiential data stream so that the local device can evaluatethe state of the remote devices, and to determine the alignment ofrecorded experiential data streams (for example, because each recordedversion of the experiential data stream may start at different points,this procedure locates some point in the experiential data stream whereeach of the remote devices have a recorded I-Frame of substantially thesame portion of the experiential data stream).

Once alignment of the I-Frames/GOPs is complete, a ‘determine start GOP’procedure 1209 determines the start GOP for the local device. A‘determine synchronization GOP’ procedure 1211 determines the expectedsynchronization GOP and sets the playback vector so that once playbackof the experiential data stream starts, the local playback vector willbe aligned with the playback vector in the other remote devices when thesynchronization GOP is reached. The first startup phase thread 1200completes through an end terminal 1213.

In some embodiments, the first startup phase thread 1200 also determineshow long it will take for the playback position to move from the startGOP to the synchronization GOP. One embodiment negotiates a point inreal-time when all the audience members' devices should be presentingthe synchronization GOP. Each of the audience members' devices thenstart presenting the experiential data stream at a real-time such thatthe playback position for each device arrives at the synchronization GOPat substantially the same point in real-time.

Other embodiments include determining a virtual audience member and thenproviding appropriate playback vector adjustments so that by the timethe synchronization GOP is reached, each audience member is aligned.This can be accomplished by using techniques similar to those describedwith respect to a ‘synchronization command’ procedure 1413 assubsequently described with respect to FIG. 14.

FIG. 13 illustrates the second phase startup thread 1300 that can beinvoked by the ‘start threads for phase 1 and 2 startup’ procedure 803of FIG. 8, initiates at a start terminal 1301 and continues to a‘synchronize time’ procedure 1303. The ‘synchronize time’ procedure 1303can determine the clock of each audience members' devices and candetermine the difference between the clocks of each audience members'devices. An additional function of the ‘synchronize time’ procedure 1303can be to determine communication latency between the audience members'devices.

A ‘select initial N I-frames’ procedure 1305 selects N content samples(such as I-frames in this MPEG-2 example) for characterization. N isselected such that I-frames from some recorded portion of theexperiential data stream will be characterized (for example, I-framesfrom 1-10 minutes of the start of the experiential data stream). Therecorded portion can be from a complete or partially recordedexperiential data stream received by transmission as well as anexperiential data stream recorded on a tangible computer-usable datacarrier (such as a DVD). The characterization information represents therecorded portion and can be used to determine the start GOP andsynchronization GOP for the local audience member's device. Thesynchronization GOP can be near the first GOP that all the audiencemembers' devices have recorded. Once the collection of I-frames isgathered, a ‘prepare characterization information’ procedure 1307evaluates the information in the I-frames/GOPs, or other contentsamples, to uniquely identify the frames as was previously described.Once these I-frames/GOPs are characterized, the ‘send characterizationinformation’ procedure 1309 sends the characterization information tothe audience members' devices and the second phase startup thread 1300completes through an end terminal 1311. The characterization informationrepresents the content sample such that by analyzing thecharacterization information from content samples substantially similarcontent samples from different experiential data streams can beidentified.

An additional technology that can be used to align the start of theexperiential data stream is enabled if appropriate content meta-data isincluded within the experiential data stream. Such content meta-data caninclude identification of the codec used to process the experientialdata stream and/or its parameters, the identification of anadvertisement portion the experiential data stream, an identification ofparental rating for particular segments of the experiential data stream.This content meta-data can simplify the replacement of particularsubject matter that has been recorded by the replacement/additionalcontent that can be obtained from the optional content provider service419, the optional program insertion provider service 433, the storagecomponent 409 or from a computer readable data carrier a accessible to,for example, the first video playback device 403. One skilled in the artwill understand that the replacement/additional content can also beassociated with the experiential data stream using techniques that canidentify a GOP or playback position in the experiential data stream.

In addition, the techniques disclosed by FIG. 12 and FIG. 13 with theirassociated text can also be applied to align a non-audience member whodesired to join the shared social space after the audience members havestarted to experience the experiential data stream. In this situation,the non-audience member's device would establish its state-exchangethreads and obtain state from the audience members. In the situationwhere the non-audience member's experiential data stream is recordedfrom the beginning, the device can determine its synchronization GOPfrom information available from the audience members' devices (makingthe formerly non-audience member now part of the shared social space),and then perform a synchronization operation with one of the otheraudience members.

In some situations, the non-audience member will only have a partiallyrecorded experiential data stream that may be missing the initial GOPsthat would include the synchronization GOP. In that circumstance, thenon-audience member's device would record over some period to establisha sufficient accumulation of GOPs, and then request the audiencemembers' devices to characterize the GOPs near their current playbackpositions. The non-audience member's device also characterizes itsrecorded GOPs and determines from the characterization information asynchronization GOP. At this point the formerly non-audience memberbecomes part of the shared social space and can synchronize with anotheraudience member.

FIG. 14 illustrates an internal control handler thread 1400 that can beused to process the local audience member's command requests. Theinternal control handler thread 1400 can be invoked by the ‘startcontrol thread’ procedure 707 of FIG. 7 and initiates at a startterminal 1401. A ‘detect local control command’ procedure 1403 detects acommand request from the local audience member. Once the command requestis detected, it can be dispatched by a ‘select’ procedure 1405 to theappropriate procedure to handle the command request and to cause thedevice to perform the operation and/or function associated with thecommand request.

If the command request is not a movement or a shared social spacerelated command (for example, show status, show program information etcetera) the internal control handler thread 1400 continues to a‘traditional non-movement or local information command’ procedure 1407that performs well known operations such as displaying local channeland/or program information. Once the ‘traditional non-movement or localinformation command’ procedure 1407 completes, the internal controlhandler thread 1400 returns to the ‘detect local control command’procedure 1403.

If the command request is a movement related command, the internalcontrol handler thread 1400 continues to a ‘standard movement command’procedure 1409 that is subsequently described. Once the ‘standardmovement command’ procedure 1409 completes, the internal control handlerthread 1400 continues to an ‘update state’ procedure 1411 that updatesthe local state of the device and can cause the local state to be sentto the remote devices. The internal control handler thread 1400continues to the ‘detect local control command’ procedure 1403.

If the command request is a synchronization related command, the‘synchronization command’ procedure 1413 is invoked to perform asynchronization operation. This procedure is subsequently described.

If the command request is a non-movement command related to the sharedsocial space, the internal control handler thread 1400 continues to a‘shared social space non-movement command’ procedure 1415 that issubsequently described.

If the command request is a “follow” related command, a ‘follow command’procedure 1417, subsequently described, can be invoked.

After the ‘standard movement command’ procedure 1409, the‘synchronization command’ procedure 1413, the ‘shared social spacenon-movement command’ procedure 1415, and the ‘follow command’ procedure1417, the internal control handler thread 1400 continues to the ‘updatestate’ procedure 1411 previously described.

The ‘standard movement command’ procedure 1409 breaks any existingfollow state and performs the requested command. These commands caninclude a pause command, a fast forward command, a shuttle command, asingle frame movement command, a timed skip command, a backward command,and a rewind command as well as other well known commands. The ‘standardmovement command’ procedure 1409 can also update the state of the localdevice to reflect the result of the command.

Another aspect of the ‘standard movement command’ procedure 1409 is thatof detecting when a command by a local audience member would be expectedto reduce the cohesion of the shared social space 523 and to inform theaudience member of this consequence. In some embodiments, theconsequence may be to remove the audience member from the shared socialspace.

The ‘update state’ procedure 1411 can record the change in local stateresulting from the ‘standard movement command’ procedure 1409, the‘synchronization command’ procedure 1413, the ‘shared social spacenon-movement command’ procedure 1415 and the ‘follow command’ procedure1417. In one embodiment, the state of the local device can be kept withthe state of the remote devices such as is shown in FIG. 15. The ‘updatestate’ procedure 1411 can also trigger continuation of the stateexchange-output thread 900 to transmit the changed state over thecontrol link to the remote devices (either directly or through aserver).

The ‘synchronization command’ procedure 1413 can provide an option forthe local audience member to condition the local device to synchronizewith audience members in the shared social space. For example, the localaudience member can select that the local device synchronize with aspecific remote audience member's device, the remote audience memberwhose playback position is closest to the Median playback position ofall the audience members in the shared social space; or can select thatthe local device synchronize to the Median playback position of all theaudience members (or any other virtual playback position value that canbe periodically computed from the playback position of the audiencemembers). Once the local device's playback position is aligned with theplayback position of the specified device, the local device can beconditioned to track the playback position of that specified device (forexample by using a follow command).

There are many ways of synchronizing a local audience member with aremote audience member. Some of these ways include a context-displayedsynchronization, a jump synchronization, and a social-contextsynchronization.

When determining the Median or Mean playback position, outlying audiencemembers or audience members who are in the process of performing amovement command (including pause) can be excluded from thatdetermination.

A local audience member's device can stop tracking a remote audiencemember's device by the local audience member invoking a movement commandsuch as pause, fast forward, rewind, et cetera. The local audiencemember can invoke a synchronization command to again align the localaudience member's playback position with that of the remote audiencemember. Once aligned, the playback positions can move out of alignmentby either viewer invoking a movement command. If the local device istracking the remote device, the local device will perform the sameoperation as the remote device (for example pausing when the remotedevice is paused et cetera). Some embodiments automatically enabletracking after synchronization. Other embodiments require an explicitcommand request to enable tracking.

Some embodiments allow that the shared social space have a masteraudience member to which all other audience members track. Oneembodiment of the synchronization algorithms are subsequently describedwith respect to FIG. 16.

The ‘shared social space non-movement command’ procedure 1415 allows thelocal audience member to enter command requests for operations that arerelated to the shared social space but that do not affect movement orcontrol the playback position. One example of such an operation canresult from the exit/enter group command used to exit or enter theshared social space. In one embodiment, this operation allows a localaudience member to exit the shared social space (for example, to causethe local audience member's device to stop distributing stateinformation to other devices). In another embodiment, this operationallows the local audience member to join a shared social space such asby terminating selected threads and restarting the terminated threadstargeted to a different shared social space with possibly a differentexperiential data stream. Still another example is an operation thatpresents the status of the shared social space to the local audiencemember.

The ‘follow command’ procedure 1417 can toggle (or set) the “followstate” (that controls the tracking operation). In addition, this commandrequest can specify which remote audience member's device to follow. Ifno specific remote device is specified, the local device canautomatically select one of the remote devices (by, for example,determining the median or mean of the locations of some or all of theaudience members within the experiential data stream and synchronizingto a remote audience member near that location; or specifying a virtualaudience member location to synchronize with). The “follow state”controls whether the local audience member's device tracks the remoteaudience member's device. In some embodiments, such as a limited socialtelevision system, a limited number of synchronization functions (suchas context-displayed synchronization and social-context synchronization)can be enabled in the context of navigating extracted experiential datastream content.

FIG. 15 illustrates a partial state representation 1500 that includesthe partial state of two of four audience members in the shared socialspace. In particular, FIG. 15 illustrates a state for Sally 1501 and astate for Jim 1503 each as seen from the local device for Sally and Jimrespectively. One skilled in the art would understand that Sue's andTom's devices would have similar state information. Such a one will alsounderstand that additional state information may be kept for each devicein addition to the state shown herein. In some embodiments, a statusdisplay (not shown) of a representation of one or more of the audiencemembers' state information can be presented to any of the audiencemembers in response to a user command request, a change of state, inresponse to an event et cetera. Thus, by receiving state informationfrom a remote audience member's device, the local audience member'sdevice can monitor the remote audience member's device. In particular,the local audience member's device can monitor the remote audiencemember's device's playback position.

A ‘device identification’ field 1505 provides information about theaudience members and their devices. The information in this field(although here represented by an audience member's name and the text“Device ID”) generally includes the address of the device associatedwith the audience member, and some human-readable text to identify theaudience member (for example, a nickname, name, handle, login, etcetera).

A ‘reference GOP’ field 1507 indicates the identification of thestarting GOP for the experiential data stream. The contents of thisfield can be determined by the ‘determine start GOP’ procedure 1209 ofFIG. 12. A ‘next GOP’ field 1509 contains a value that indicates the GOPof the next GOP (based off of the Reference GOP) that is expected to beviewed.

A ‘follow state’ field 1511 contains state indicators such as “followed”that indicate that the device of some other audience member is trackingyour local device; “following” that indicates that audience member istracking some other audience member; “follow X” that indicates that thelocal device is tracking X's device. One possible consequence of being“followed” is that the speed of a rewind or fast forward operation maybe adjusted to better allow the audience member(s) that are following toefficiently align the playback positions.

A ‘synchronization state’ field 1513 can be used to indicate thesynchronization state of a local or remote device (including the type ofsynchronization, see the subsequent discussion related to FIG. 16). A‘playback rate’ field 1515 can be used to indicate the playback rate ofa device.

A ‘median virtual member’ record 1517 can be used to indicate thecurrent playback position in the experiential data stream that would bethe middle (median, mean or other calculated value) of the shared socialspace. The value in this field can be periodically updated based on thevalues of the other entries in the ‘next GOP’ field 1509. In someembodiments, audience members who are sufficiently far from thecalculated value may be dropped and the value recalculated.

FIG. 16 illustrates a synchronize with remote audience member process1600 that can be used in an embodiment of the social television system400. Some of the synchronization modes can include a context-displayedsynchronization, a jump synchronization, and a social-contextsynchronization. The modes can be specified by the local audience memberusing any interface to the local audience member's device.

The synchronize with remote audience member process 1600 can beinitiated by the ‘synchronization command’ procedure 1413 responsive toa local audience member command request or, if invoked by otherprocedures, as needed. The synchronize with remote audience memberprocess 1600 starts at a start terminal 1601 and continues to a‘determine synchronization options for remote audience member’ procedure1603 that determines whether the synchronization command request is fora context-displayed synchronization, whether the synchronization is fora jump synchronization, whether the synchronization is for asocial-context synchronization, or for some other type ofsynchronization. A ‘determine time to alignment with remote audiencemember’ procedure 1605 determines the amount of real-time to take toalign the local audience member's playback position with the targeted(real or virtual) remote audience member's playback position. The‘determine time to alignment with remote audience member’ procedure 1605can be responsive to the synchronization mode.

If the synchronization option is for a jump synchronization, the localaudience member will be synchronized as quickly as possible with theremote audience member by effectively setting the local audiencemember's playback position to be that of the remote audience member'sexpected playback position at the completion of the jump synchronizationas adjusted by the expected time required to perform the synchronizationand accounting for communication latency. This synchronization mode canresult in a loss of context of both the experiential data stream and thecommentary audio clips within the shared social space because a portionof the experiential data stream can be skipped and any recordedcommentary audio clip from a remote audience member associated with theskipped portion will not be presented.

If the synchronization option is for a context-displayedsynchronization, the ‘determine time to alignment with remote audiencemember’ procedure 1605 can be invoked and can determine from the partialstate representation 1500 the respective playback positions of theremote and local audience members, determine the delta time between theplayback positions, select a synchronization interval, and determine theadjustment to the local audience member's playback vector (and/or insome embodiments to the remote audience member's playback vector). Acontext-displayed synchronization mode adjusts the local audiencemember's playback vector from the nominal playback vector (faster orslower) such that, over some synchronization interval, the localaudience member's and the remote audience member's playback positionbecome aligned (after which time the local audience member's playbackvector can be set equal to the remote audience member's playback vector,or returned to its nominal value). The adjusted playback vector can bedetermined from a viewing distance (the separation in the playbackpositions) and selection of a suitable synchronization interval withinwhich to achieve alignment of the devices. The synchronization intervalcan be generally selected such that the local audience member cancomprehend the portion of the experiential data stream that is beingpresented using the adjusted playback vector. The synchronizationinterval can be heuristically determined and represented as a functionof the viewing distance and the playback vector of the remote audiencemember's device. The audio (both the audio from the experiential datastream or replacement/additional content, as well as any transitoryaudio or commentary audio clip) associated with the portion of theexperiential data stream that will be presented at other than nominalspeed can be pitch compensated accordingly. Thus, the remote audiencemember can synchronize with the remote audience member while still beingexposed to the context that would be missed by a jump synchronization.

If the synchronization option is for a social-context synchronization,the ‘determine time to alignment with remote audience member’ procedure1605 can again determine from the partial state representation 1500 therespective playback positions of the remote and local audience members,and determine the local audience member's playback vector value so thatthe playback position of the local audience member and the remoteaudience member will become aligned after some unspecified interval. Asocial-context synchronization is one that uses a faster playback vector(in some embodiments, resulting in a “fast-forward” playback vector) foradvancing the experiential data stream of the local audience memberuntil proximity to the audio trigger position of a commentary audio clipis detected. When this occurs, the local audience member's playbackvector can be returned to the nominal vector so that the local audiencemember can experience the content in the experiential data stream thatengendered the pending commentary audio clip. The local audiencemember's playback vector will remain nominal, and all the commentaryaudio clips will be reproduced until some period after the commentaryaudio clips related to that portion of the experiential data streamcompletes. At this point, the local audience member's playback vectorwill again be increased to advance the experiential data stream untilthe next the audio trigger position is approached, the local audiencemember cancels the synchronization, or the local audience member'splayback position aligns with the remote audience member (at whichpoint, the playback vector can be returned to nominal). During thesynchronization period, the remote audience member's playback positioncontinues to advance. Thus, this synchronization mode periodicallydetermines the distance between the local audience member's and remoteaudience member's playback positions and terminates the synchronizationwhen they become aligned the same.

The social-context synchronization differs from the context-displayedsynchronization in that in the social-context synchronization theimportance of the social communication within the shared social space isgiven priority over the speed of synchronization whereas in thecontext-displayed synchronization the playback vector is simplyincreased (for both the experiential data stream orreplacement/additional content, and the commentary audio clips) and thusthe local audience member will not fully experience the context of theexperiential data stream with the commentary audio clips.

The processes used by either of the social-context synchronization orthe context-displayed synchronization can also be used by a localaudience member who is not a member of the shared social space. Thus, ifcommentary audio clip group is purchased or has already been recordedfrom audience members in a previous shared social space, the localaudience member can playback of the experiential data stream with theassociated commentary audio clips. The local audience member can fastforward through the experiential data stream using one of the disclosedsynchronization modes to experience the experiential data stream and itscommentary audio clips. In this situation, the local audience member'sdevice has no need to maintain state from a remote audience member'sdevice, or to achieve alignment. An extracted-experiential data streamof the experiential data stream can be played back by the local audiencemember to experience the experiential data stream and its commentaryaudio clips without fast forwarding because the extracted-experientialdata stream only contains content that develops the shared social space.The local audience member can also select which commentary audio clipsor other commentary clips (individually or by group) are of interest. Ifthe local audience member made a selection of commentary clips, thelocal audience member can fast forward through theextracted-experiential data stream using one of the disclosedsynchronization modes to experience the extracted-experiential datastream within the context of the selected commentary clips to obtain alimited experience of the shared social space.

Each of the synchronization modes stores a different state in the‘synchronization state’ field 1513.

A ‘set follow state’ procedure 1607 can set the ‘follow state’ field1511 and the ‘playback rate’ field 1515 for the local audience member toindicate which real or virtual remote audience member is to be (or iscurrently) tracked, and if the synchronization mode is acontext-displayed synchronization, will set the ‘synchronization state’field 1513 and the ‘playback rate’ field 1515 to the appropriate values.This change in state will be propagated to at least the deviceassociated with remote audience member. Once the synchronization mode isdetermined and the local state adjusted, an ‘adjust playback vector orposition’ procedure 1609 then starts the synchronization process inaccordance with the previously described synchronization modes. Thestate for the local audience member's device can be updated accordingly.

Next, a ‘wait for event’ procedure 1611 waits for a condition such aswhen the playback position of the local audience member and the followedremote audience member become aligned. One skilled in the art wouldunderstand that there are many different ways to determine theoccurrence of this event (including polling, occurrence of ahardware-generated event, et cetera). Once the playback position of thefollowed remote audience member and local audience member are aligned an‘adjust playback vector’ procedure 1613 adjusts the payback vector (ifrequired) to match that of the followed remote audience member. An‘adjust local state’ procedure 1615 adjusts the local state such as byresetting the ‘synchronization state’ field 1513, the ‘playback rate’field 1515 if need be, and possibly the tracking state. The synchronizewith remote audience member process 1600 completes through an endterminal 1617.

The ‘wait for event’ procedure 1611 can also detect the audio triggerposition of an upcoming commentary audio clip from a remote audiencemember or purchased commentary audio clip for the experiential datastream such that the ‘adjust playback vector’ procedure 1613 can returnthe playback vector to nominal to effect the social-contextsynchronization mode. Then, the synchronize with remote audience memberprocess 1600 returns to the ‘wait for event’ procedure 1611 to wait forthe completion of the presentation of the commentary audio clip. Oncethe commentary audio clip is presented, an event can be triggered anddetected such that the ‘adjust playback vector’ procedure 1613 canresume the synchronization that was interrupted by detection of theaudio trigger position of the commentary audio clip.

Further, the ‘wait for event’ procedure 1611 can wait for other eventsthat result from the local audience member invoking a command requestthat cancels the synchronization or is inconsistent with thesynchronization.

In addition to the jump synchronization, the context-displayedsynchronization, and the social-context synchronization, the inventorshave developed a wait-for synchronization that can be used to pause orslow down playback until a remote audience member's playback positioncatches up to the local audience member's playback position.

The local audience member can interact with the local device using anymethod possible including a controller, a computer, a web page,Bluetooth™, Obje™ technology et cetera multiple button push, menuselection, infrared devices, and the like. In addition, eye-trackingtechnology can be used from the images obtained from the video camera435. Further presence information as to who belongs to the localaudience members can be sent to the remote audience members from thevideo camera 435 through any of the possible communication pathsestablished by the social television system 400.

FIG. 17 illustrates a control panel 1700 that can be used by a localaudience member to control their participation in the shared socialspace or to control other aspects of the disclosed technology whether ornot applied to the shared social space. The control panel 1700 can beincorporated with the first video playback device 403 and/or the controldevice 441. The control panel 1700 interacts with the local audiencemember's playback device to cause the playback device to perform enabledfunctions (for example, by issuing command requests to the playbackdevice, or by conditioning electronics or other circuitry or procedureswithin the playback device). These command requests can be issued by,for example, using buttons on the control panel 1700 for the first videoplayback device 403, using a separate control device (similar to a DVDor DVR remote control), can be implemented using a general device orcomputer that can be programmed to interact with the local audiencemember's device through numerous mechanisms such as the Internet,Infrared protocols, Bluetooth™ protocols, Obje™ protocols, and anytechnology for providing communications between a remote control deviceand the device the remote is to control (including web interfaces).

One embodiment of the control panel 1700 comprises a housing 1701containing, for example, a power supply, control circuitry,communication circuitry et cetera (all not shown), and structured tosupport user selectable controls (USC) such as a first standard USC 1703that could be a power button, a second standard USC 1705 that could be anumber button, an N^(th) standard USC 1707 that could be any USC thatwould commonly be on a remote control device for an entertainment systemor otherwise. A ‘synchronize’ USC 1709 can be one mechanism that, whenactivated, issues a command for initiating the synchronization processand/or the tracking process that is described herein and for which oneembodiment has been described, for example, with respect to FIG. 16. The‘synchronize’ USC 1709 can also be used to replay the experiential datastream and associated commentary audio clips on the audience member'sdevice using, for example, a context-displayed synchronization or asocial-context synchronization (even if the audience member is notsharing the shared social space with a remote audience member). A‘follow’ USC 1711 can be one mechanism that, when activated, issues acommand for initiating the follow process that enables tracking of aremote audience member's playback position that was previously describedand for which one embodiment has been described with respect to FIG. 6.A ‘group’ USC 1713 can be one mechanism, when activated, for issuing acommand for initiating, entering or exiting the shared social space andcan be used to initiate the ‘join guests’ procedure 603 previouslydiscussed with respect to FIG. 6.

User selectable controls that invoke commands that can invoke the‘update state’ procedure 1411 when in a shared social space, are termedsocial-space related user selectable controls. Some of the social-spacerelated user selectable controls also control capabilities disclosedherein that can operate outside of the shared social space.

In addition, USCs or control options can be used that allow the localaudience member to switch to nominal speed playback while in asynchronization mode. This can be accomplished by a command issued by acancel synchronization USC, or by the user activating the ‘synchronize’USC 1709 to issue a synchronization command when the device is alreadyin a synchronization state. In addition, the inventors contemplatecommand options that allow the local audience member to condition thesynchronization modes (or an increased playback mode, fast forward) suchthat as the audio trigger position of a commentary audio clip isapproached, the speed of presentation of the experiential data streamwould be automatically changed to nominal speed so that the commentaryaudio clip can be presented with the supporting portion of theexperiential data stream. Once the commentary audio clip is complete,presentation of the experiential data stream could then automaticallyrevert to the previous playback speed, or automatically re-invoke thesynchronize with remote audience member process 1600 to continue thesynchronization process from the current state. One skilled in the artwill understand that commands issued responsive to some of the standardUSCs may affect other audience members. For example, if a followedaudience member pauses or fast forwards his/her local device, thefollowing audience members' local devices will do the same. Further somelocal commands over some period of time may reduce the cohesion of theshared social space and the audience members can be removed from theshared social space.

Another aspect of a social television system is that of coordinatingtransitory audio (and possibly video) from remote audience members withthe audio from the experiential data stream. The problem is that formany types of experiential data streams (for example, types oftelevision program) comments from the audience members need to bepresented at non-disruptive positions in the experiential data stream.Otherwise, other audience members can be annoyingly distracted from theexperiential data stream (note that this is not an issue with silentvideo of the remote audience member whether this video is presented asPicture in Picture (PIP) or on a separate device). One skilled in theart will understand that there are additional issues that result fromthe possibility of some of the audience members having playbackpositions significantly out of alignment with the other audiencemembers.

Commentary audio clips from remote audience members having playbackpositions sufficiently out of alignment may not be presented to thelocal audience member (however, the transitory audio can still berecorded as a commentary audio clip for the local audience member andindexed to the audio trigger position—that is, the playback position ofthe remote audience member's device at the time of the utterance). Inthe circumstance where the remote audience member is in advance of thelocal audience member, the remote audience member's audio can berecorded for presentation when the local audience member reaches thecorresponding audio trigger position. If the commentary audio clip isreceived and indexed to a playback position prior to the local audiencemember's current playback position, the commentary audio clip will bestored, but will be not be presented unless the local audience memberrewinds the experiential data stream or otherwise traverses the portionof the experiential data stream containing the audio trigger position.In addition, the local audience member's device can record all audio atthe playback position when the audio was uttered (both locally detectedaudio as well as audio detected at the remote devices). Thus, asubsequent playback of the experiential data stream optionally caninclude all the audio from all the audience members' utterances while inthe shared social space. Thus, an enhanced recording of the sharedsocial space can be presented to each of the audience members.

Audio from both the local and remote audience members (separately orcombined) can be recorded with respect to the playback position of thespeaker such that subsequent playback of the experiential data streamwill include all or selected commentary audio clips appropriatelysequenced to the experiential data stream (this allows a subsequentplayback to present commentary audio clips made by others whose playbackposition when the uttered audio was recorded was behind the playbackposition of the local audience member during the initial experience ofthe shared social space). A copy of all or a portion of the commentaryaudio clip can be separately stored on a tangible computer usable datacarrier or transmitted across a network. The local audience member canselect whether to separately store only his/her commentary audio clips,all commentary audio clips, or any selection of commentary audio clipsselected by audience member and/or portion of the experiential datastream.

There are other embodiments for presenting audio between the audiencemembers. In one embodiment, characterized by each of the audiencemembers who are in a conversation being aligned in the experiential datastream, the audio from each of these audience members can be presentedto the other audience members in real-time. This embodiment can also beused with conversation finder technology as is taught in the previouslyincorporated-by-reference published United States patent applications tosupport side conversations between some of the local audience memberswith some of the remote audience members.

In one embodiment subsequently described with respect to FIG. 18,commentary audio clips can be downloaded (or otherwise delivered) to thelocal audience member's device and associated with the experiential datastream. Thus, a local audience member can have his/her audio stored andindexed to the experiential data stream and can then share his/hercommentary audio clips with others. In addition, the commentary audioclips can be provided for sale.

Further, commentary audio clips can be associated with the experientialdata stream during the same presentation of the experiential data streamand can be individually conditioned (such as by using controls asdescribed with respect to FIG. 17) to be presented, and for thosecommentary audio clips that are to be presented having an audio durationthat overlaps other commentary audio clips, can be conditioned to bepresented in simultaneous, sequential, instant-replay, or other modes.

Each commentary audio clip in a simultaneous presentation mode can bepresented in real-time with the experiential data stream. Thus, multiplecommentary audio clips can be simultaneously presented as each istriggered as the playback position approaches an audio trigger positionin the experiential data stream.

Each commentary audio clip in a sequential presentation mode can bepresented in order (the order can be specified by the local audiencemember) such that when the playback position reaches the audio triggerposition, all simultaneous mode commentary audio clips (if any) and thefirst ordered sequential commentary audio clip can be presented followedby the second ordered sequential commentary audio clip, then the thirdet cetera. While the sequential commentary audio clips are beingpresented, any simultaneous commentary audio clip can be also presentedat the audio trigger position. This allows sequential presentation ofcommentary audio clips related to a playback position in theexperiential data stream. In one embodiment, the length of the combinedcommentary can be determined, and the playback vector can be adjusted sothat some or all of the sequential commentary audio clips aresequentially presented during the same period in the experiential datastream time dimension as the time period to required to present thelongest sequential commentary audio clip alone using the nominalplayback vector.

For commentary audio clips having an instant-replay presentation mode,the first ordered instant-replay commentary audio clip can be presentedalong with any commentary audio clips in simultaneous presentation modewhile the experiential data stream can be presented at nominal speed.After the first ordered instant replay commentary audio clip ispresented, the experiential data stream can be re-wound to the playbackposition of the first instant-replay commentary audio clip and thesecond ordered instant replay commentary audio clip can be presented(but not the first) as well as optionally replaying the simultaneouscommentary audio clip.

In some embodiments, audio from each of the audience members can beprocessed to recognize whether the audio contains non-verbal vocalutterances such as laughter, sobbing, groaning, et cetera. This audiocan be merged to make a commentary audio clip group containing theseutterances from the audience members in the shared social space, thus,creating a commentary audio clip group that is equivalent to a privatelaugh-track with utterances recorded from audience members participatingin the shared social space. The commentary audio clip can be processedto detect non-verbal vocal utterances and classified accordingly (forexample when the commentary audio clip is created or when the commentaryaudio clip is indexed to the experiential data stream).

One skilled in the art will understand that the local audience membercan specify options as to which of the commentary audio clips (based onaudience member identification) is to be presented and can selectivelydisable commentary audio clips as desired.

FIG. 18 illustrates a commentary audio clip presentation process 1800that initiates at a start terminal 1801 responsive, for example butwithout limitation, to a timer, other event, or a periodic thread. Onceinitiated, the commentary audio clip presentation process 1800 continuesto a ‘detect pending commentary audio clip’ procedure 1803 thatdetermines the playback position of the experiential data stream andlocates upcoming audio trigger position(s) that will trigger imminentlypending commentary audio clip. Next, a ‘dispatch’ procedure 1805 scansthe index to the imminently pending commentary audio clip to determineits presentation mode. If the commentary audio clip is in simultaneousmode, the commentary audio clip presentation process 1800 continues to a‘present simultaneous commentary audio clip’ procedure 1807 that cues upthe commentary audio clip for presentation when the playback positionreaches the location specified in the commentary audio clip. Then thecommentary audio clip presentation process 1800 completes through an endterminal 1809 (although the process can also repeat until all imminentlypending commentary audio clips are handled or by subsequent eventpostings). Techniques similar to those using the ‘detect pendingcommentary audio clip’ procedure 1803 that determines the playbackposition of the experiential data stream and locates upcoming audiotrigger position(s) that will trigger imminently pending commentaryaudio clip(s) can also be used with commentary clip(s) associated withcommentary clip trigger position(s) as commentary audio clips are asubset of the commentary clip.

At the ‘dispatch’ procedure 1805, if the commentary audio clip is insequential presentation mode, the commentary audio clip presentationprocess 1800 continues to a ‘detect and order overlapping sequentialcommentary audio clip’ procedure 1811 that scans the indices to theimminently pending sequential commentary audio clips to determinewhether there is an overlap between the first such commentary audio clipand a subsequent commentary audio clip. If no overlap exists, thecommentary audio clip can be processed by the ‘present simultaneouscommentary audio clip’ procedure 1807 (the transfer of such a sequentialcommentary audio clip to the ‘present simultaneous commentary audioclip’ procedure 1807 is not shown). If an overlap occurs between twoimminently pending sequential commentary audio clips, the ‘detect andorder overlapping sequential commentary audio clip’ procedure 1811determines the time required to present each of the overlappingsequential commentary audio clips. An ‘optional adjust playback vector’procedure 1813 can adjust the playback vector so that the sequentialplaying (one after another) of all of the overlapping commentary audioclips in real-time will span the same period as the longest of thecommentary audio clip in the experiential data stream time (for example,if there are two overlapping commentary audio clips, determining thetime required to sequentially play both in non-overlapping fashion,adjusting the playback vector to slow the presentation of theexperiential data stream such that both commentary audio clips arepresented during the presentation of the slowed experiential datastream). The ‘optional adjust playback vector’ procedure 1813 and the‘detect and order overlapping sequential commentary audio clip’procedure 1811 coordinate to adjust the playback position for thecommentary audio clips such that a subsequent commentary audio clipstarts following the completion of the previously presented commentaryaudio clip. The ‘optional adjust playback vector’ procedure 1813 isoptional in that it allows sequential commentary audio clips to bepresented while still presenting the corresponding content from theexperiential data stream (although in a slow speed fashion).Alternatively, the playback vector need not be adjusted so long as theoverlapping commentary audio clips are serialized.

A ‘sequentially present commentary audio clips’ procedure 1815 thenpresents the commentary audio clips (one skilled in the art willunderstand that the presentation of the commentary audio clips canaccomplished in a plethora of ways). Once the commentary audio clips arepresented, an ‘optional resume playback vector’ procedure 1817 restoresthe playback vector to the nominal speed of the experiential data streamif required and the commentary audio clip presentation process 1800completes via the end terminal 1809.

Commentary audio clips conditioned to be presented in simultaneous modecan also be presented during the period that sequential commentary audioclips are being presented.

At the ‘dispatch’ procedure 1805, if the commentary audio clip is to bepresented in instant-replay presentation mode, the commentary audio clippresentation process 1800 continues to a ‘count overlappinginstant-replay commentary audio clips’ procedure 1819 that scans theimminently pending indices to commentary audio clips to determinewhether there is an overlap between the first such instant-replaycommentary audio clip and a subsequent commentary audio clip. If nooverlap exists, the commentary audio clip can be processed by the‘present simultaneous commentary audio clip’ procedure 1807 (thetransfer of such a sequential commentary audio clip to the ‘presentsimultaneous commentary audio clip’ procedure 1807 is not shown). If anoverlap occurs between two imminently pending instant-replay commentaryaudio clips, the ‘count overlapping instant-replay commentary audioclips’ procedure 1819 counts the number of such overlaps and can storethe playback position of the first occurring commentary audio clip andthe playback position corresponding to the end of the last commentaryaudio clip. A ‘present first overlapping commentary audio clip’procedure 1821 then causes the first in time commentary audio clip to bepresented. During the presentation of this instant-replay commentaryaudio clip simultaneous commentary audio clips are also presented. Inone embodiment, the experiential data stream can be presented until theplayback position corresponding to the end of the last instant-replaycommentary audio clip is reached. This allows all the simultaneouscommentary audio clips to be presented during the first iteration of theinstant-replay period (in other embodiments the simultaneous commentaryaudio clips can be presented on each iteration, or each iteration canterminate at the end of the presently presented commentary audio clipwith or without presentation of the simultaneous commentary audioclips).

For the embodiments that do not allow simultaneous commentary audioclips to be presented on subsequent iterations, a ‘disable simultaneouspresentation of commentary audio clip’ procedure 1825 disables thecapability. A ‘for each remaining instant-replay commentary audio clip’iterative procedure 1827 then iterates each instant-replay commentaryaudio clip to a ‘rewind’ procedure 1829 that rewinds the experientialdata stream to (depending on the embodiment) the saved playback position(so that each instant-replay iteration starts at the same location), orrewinds back to before the playback position of the next iteration. A‘present instant-replay commentary audio clip’ procedure 1831 thatoperates substantially the same as the ‘present first overlappingcommentary audio clip’ procedure 1821 then presents the iteratedinstant-replay commentary audio clip and the commentary audio clippresentation process 1800 returns to the ‘for each remaininginstant-replay commentary audio clip’ iterative procedure 1827 tocontinue iteration of the instant-replay commentary audio clips.

When all the instant-replay commentary audio clips have been iterated,the commentary audio clip presentation process 1800 continues to an‘enable simultaneous presentation of commentary audio clip’ procedure1833 that re-enables presentation of simultaneous commentary audio clipsif it was disabled by the ‘disable simultaneous presentation ofcommentary audio clip’ procedure 1825 and the commentary audio clippresentation process 1800 completes through the end terminal 1809.

In yet another embodiment, when an audio is received from anotheraudience member the receiving system can determine the best place toinsert the audio into the experiential data stream instead of insertingthe audio at the playback position corresponding to where the commentaryaudio clip or transitory audio was uttered.

FIG. 19 illustrates an audio insertion process 1900 that can be used todetermine where in the experiential data stream to insert transitoryaudio received from a remote audience member or local audience member.The audio insertion process 1900 can be used alone or in conjunctionwith other audio processing disclosed herein. The audio insertionprocess 1900 can be part of an event process or can be invoked through areceive audio thread (not shown) that receives and assembles thetransitory audio stream from the remote audience member (or captures anddigitizes the transitory audio from an utterance of a local audiencemember). The audio insertion process 1900 initiates at a start terminal1901 and continues to a ‘look-ahead’ procedure 1903.

The ‘look-ahead’ procedure 1903 analyzes a later portion of theexperiential data stream that is ahead of the local playback position inthe time dimension to find candidate portions of the experiential datastream that may be suitable for placement of the received transitoryaudio. Examples of such portions include quiet areas, areas containingcommercials, or portions of the experiential data stream that haveheuristically determined characteristics.

A ‘detect audio length’ procedure 1905 determines the durationcharacterization of the transitory audio or expected durationcharacterization (for example, by looking at the history of audio fromthat remote audience member). The ‘detect audio length’ procedure 1905can also determine other characterizations of the transitory audiostream (such as loudness, type of utterance, et cetera) and can storethe transitory audio (such that it can be converted into a commentaryaudio clip that has an audio trigger position associated with theexperiential data stream).

A ‘determine cost of insertion’ procedure 1907 then computes a minimumcost for each candidate portion that includes a charge for delaying thetransitory audio from the playback position where it was uttered, acharge for compressing the duration of the transitory audio, a chargefor having overlap between the transitory audio and significant audiofrom the experiential data stream, and, in some embodiments, charges foroverlapping transitory audio from two of the remote audience members,having transitory audio from multiple remote audience members presentedout of order, or having similar conflicts with commentary audio clip.

One result of the ‘determine cost of insertion’ procedure 1907 can be todetermine an audio trigger position in the experiential data streamwhere the commentary audio clip version of the transitory audio is to bepresented to the local audience member. Next an ‘insert audio’ procedure1909 stores the transitory audio as a commentary audio clip (if notalready stored) indexed to the experiential data stream by an audiotrigger position. The audio insertion process 1900 then completesthrough an end terminal 1911.

Another aspect of the herein disclosed technology is the business ofproviding commentary audio clips and replacement/additional content toan audience member. One such business offers commentary audio clips forsale to an audience member who then purchases the commentary audio clip.The commentary audio clips can be delivered over the network (a computerusable data carrier), on tangible computer usable data carrier, or viaother techniques known in the art. Another such business providesreplacement and/or additional advertisements for use when presenting theexperiential data stream. The advertisers would pay for the number ofplacements or will pay using any negotiated metric known in the art.Another such business provides replacement content for portions of theexperiential data stream to allow selected GOPs or groups of GOPS in theexperiential data stream to be replaced by altered content toaccommodate parental controls, audience member's rating preferences,and/or to provide fine control over particular aspects of thepresentation of the experiential data stream as regards sexualsituations, violence, language and/or other aspects of the content.

FIG. 20 illustrates a process for providing commentary audio clip and/orreplacement/additional content 2000. A ‘contract for commentary audioclip’ step 2001 arranges for the creation of commentary audio clipsrelated to the experiential data stream by appropriate talent (thecreator). The ‘receive, store, and classify content’ step 2003 receivesthe commentary audio clip or any other content (such as that created asa result of the ‘contract for advertisement placement’ step 2005 or the‘contract for replacement/additional content’ step 2007) eitherelectronically (over a network) or via a tangible computer-usable datacarrier, stores the content in a networked computerized library orserver system so that it can be made available to an audience member'sdevice, and classifies the content as to its type, contractualobligations, and et cetera. A ‘make content available’ step 2009 thenmakes the content available to the public for purchase. A ‘receiverequest for content’ step 2011 receives a request at a networkedcomputer system for the content either from the audience member'splayback device, by an electronic order or by a paper order subsequentlyentered into the networked computer system. A ‘provide requestedcontent’ step 2013 then provides the content to the customer. Thecontent can be provided to the customer from the networked computersystem over any computer usable data carrier including a network and/ortangible computer usable data carrier. The content can be conditionedusing digital rights management to only be presented in conjunction withthe corresponding experiential data stream for which the content wascontracted. The networked computer system records the transactioninformation. Once the content is provided, the relevant entity can bebilled for the delivery by a ‘charge for providing content’ step 2015.

In some embodiments, the optional program insertion provider service 433can maintain demographic, preference, and/or use information about theaudience members in the shared social space. With this information, theoptional program insertion provider service 433 can providereplacement/additional content the selection of which is targeted to theviewing habits, personal preferences, or viewing habits of individualaudience members as well as towards the shared social space itself.

Communication between local audience members and remote audience memberscan also be accomplished using a keyboard or other mechanism to providecommunication between the audience members.

The inventors have realized that information other than commentary audioclips can be used with the experiential data stream. Thus, examples ofcommentary clips include a commentary audio clip, an audio clip, a videoof an audience member, an audio of an audience member, text from a chatsession with an audience member, text resulting from application ofspeech recognition to an utterance by an audience member, an animationprovided by an audience member, and a still image that can be recordedand indexed to trigger position in the experiential data stream. Theinventors have also realized that it is advantageous to be able toextract content from the experiential data stream that is related to oneor more commentary clips as an extracted-experiential data stream andthus capture and distill the social context related to the experientialdata stream. The extracted-experiential data stream can be subsequentlypresented to close a loop of interaction. The extracted-experientialdata stream can also optionally include replacement/additional content.

The still image can include a frame from a video of a local audiencemember, or any other still image accessible by the social televisionsystem. The animation can be a series of separate still images(slideshow), a video clip (such as of the local audience member), acartoon, etc.

One aspect of the technology is a system (or an extension to a socialtelevision system) that produces a presentation of extracts of anexperiential data stream and/or commentary that can be shared with aresponse-reviewer. Aspects of the technology generate predefined andconstrained presentations that can be shared and/or reviewed by theresponse-reviewer (either remotely or in copresence) using less complexdevices than that of a social television system (for example, by using alimited social television system, a DVD or other content-appropriateplayback device) to create a compelling feeling of “shared experience”on the part of the response-reviewer.

FIG. 21 illustrates a first content extraction process 2100 that can beused to store an extracted-experiential data stream that includes acommentary clip along with portions of the experiential data stream thatare associated with the commentary clip. A ‘locate commentary clip’procedure 2101 locates a commentary clip associated with theexperiential data stream. A ‘determine commentary clip extent’ procedure2103 then determines content extents for the experiential data streamthat are related to the commentary clip (for example, by determining thestarting and ending positions (on the time axis) of the commentaryclip).

Once the content extents are determined an ‘extract content relevant tocommentary clip’ procedure 2105 extracts content from a portion of theexperiential data stream that is related to the commentary clip (asdetermined by the content extents). The extracted experiential datastream content and/or the commentary clip can then be stored by a ‘saveextracted content’ procedure 2107 as an extracted-experiential datastream. The ‘save extracted content’ procedure 2107 can store theextracted content of the experiential data stream and/or the commentaryclip separately, or combined. Furthermore, the ‘save extracted content’procedure 2107 can “store” the extracted-experiential data stream bytransmitting the extracted-experiential data stream over a network forstorage on a network device or displayed as received by a networkdevice. In addition, the extracted-experiential data stream can bestored on a tangible data storage media for later playback by a socialtelevision system or a limited social television system or in a formatfor playback by any video playback device.

By allowing one or more response-reviewers to display the informationstored by the ‘save extracted content’ procedure 2107, multiple audiencemembers can close the loop of interaction and create a compellingfeeling of “shared experience” on the part of the response-reviewer.

FIG. 22 illustrates a second commentary extraction process 2200 thatillustrates a more advanced process than that shown in FIG. 21 that canbe used to store commentary clips with the associated portions of theexperiential data stream as the extracted-experiential data stream. A‘select enabled commentary clips’ procedure 2201 selects whichcommentary clip(s) is(are) to be included. A ‘determine extent ofcommentaries’ procedure 2205 analyzes overlapping commentary clips (inlight of their presentation modes) in a manner similar to that describedwith respect to FIG. 18 and determines the content extents accordingly.The commentary clips can be from a local audience member or a mixture ofcommentary clips from the local audience member, a remote audiencemember and/or purchased commentary clips.

Once the content extents are determined, a ‘tag content relevant toextents’ procedure 2207 tags the contents of the experiential datastream that will be extracted and a ‘tag commentary clips’ procedure2209 tags the commentary clips that will be extracted. Once the contentand commentary clip (and any other information is tagged), a ‘storetagged information’ procedure 2211 stores the tagged information as theextracted-experiential data stream.

Some embodiments allow a user to remove a tag related to previouslyenabled commentary clips, or to mark commentary clips as not to beenabled. The commentary clips can be enabled on a social televisionsystem by, for example but without limitation, a user activating a userselectable control to manually command the social television system toenable the commentary clip, automatically enabling the commentary clipresponsive to a user's activation of a user selectable control thatcommands the social television system to perform a primary operation(for example, “skip back eight seconds”) and that also enables thecommentary clip, providing a command to a social television system toenable all (or some number) of commentary clips; and the socialtelevision system automatically analyzing the commentary clips and/ortransitory audio to determine viewer engagement and enabling thecommentary clips accordingly, or (where the social television systemcaptures video of the viewers) performing video scene analysis todetermine the social engagement of the viewers and enabling commentaryclips accordingly. One embodiment detects the local audience members'periods of speech and the stored information is selected from theexperiential data stream extending from one minute before and ending oneminute after such periods of speech. Of course, such automaticmechanisms can be supplemented by optional manual or semi-automaticmechanisms, as well as with optional semi-automatic editing operations(such as deletion of, or disablement of, commentary segments that thelocal audience member does not wish to be shared).

The technologies illustrated in FIG. 21 and FIG. 22 can be implementedby a social television system, a limited social television system, byspecial purpose electronics in a video system or computer, or by anappropriately programmed general purpose computer. These technologiescan be provided by a service that accepts the commentary and assemblesthe presentation.

Techniques taught related to FIG. 18 can be used by the ‘extract contentrelevant to commentary clip’ procedure 2105 and the ‘determine extent ofcommentaries’ procedure 2205. Thus, where commentary clips overlap, theextracted content can be stored responsive to processing similar to thatdescribed with respect to the commentary audio clip presentation process1800 of FIG. 18 and the response-reviewer will experience thepresentation in a similar manner as if experienced on a socialtelevision system.

The ‘determine extent of commentaries’ procedure 2205 (as well as the‘determine commentary clip extent’ procedure 2103) also can include thecapability of extending the leading extent, the trailing extent or bothby a predetermined amount of time (the leading and trailing extentextensions need not be by the same amount). It also has the capabilityof: extending the leading extent, the trailing extent or both to one ormore video scene transitions within the experiential data stream;extending the leading extent, the trailing extent or both to one or moreconversation transition(s) (for example by using techniques similar tothose described with respect to the ‘determine social/view balance’procedure 1113 to analyze audio from within the experiential data streamor transitory audio from the viewer(s)); and extending the leadingextent, the trailing extent or both to a social engagement transitiondetermined from analyzing video data of one or more local audiencemembers to determine a social engagement transition.

The ‘store tagged information’ procedure 2211 and the ‘save extractedcontent’ procedure 2107 can store the extracted-experiential data streamand/or the commentary clips separately, or combined. Furthermore, theextracted-experiential data stream can be stored by transmitting it overa network for storage on a network device or displayed as received by anetwork device. In addition, the extracted-experiential data stream canbe stored on a tangible data storage media for later playback by asocial television system, a limited social television system or by anyvideo playback device (for example, a handheld computer, game console,phone, computer, DVD player, video camera in playback mode, etc.).

The extracted-experiential data stream and/or the commentary clips canbe stored in a format that allows multiple audio tracks to be associatedwith the extracted-experiential data stream or stored in a form/formatthat can be used with a social television system or a limited socialtelevision system. Further, the extracted-experiential data streamand/or commentary clip can be combined within one or more presentationtracks or layers, can be individually stored on separate presentationtracks or layers for later recombination (such as by the socialtelevision system 400 or a limited social television system), can bestored consistent with a program track and multiple audio tracks, orstored in any appropriate format known to one skilled in the art. Forexample, the extracted-experiential data stream data can be layered toselectively enable presentation of one or more layers (for example anextract layer, Bob's commentary layer, June's commentary layer, or somecombination) during playback. Each layer or layer combination can becreated to be independent of the other layers, or multiple layers can beprovided and selected during playback by an appropriately enabledplayback device.

The extracted-experiential data stream can include one or more of audiodata, video data, single image data, audio effect data, and video effectdata. The still image data can be determined by usingscene-change-detection algorithms that are well known in the art. Thevideo effect data can be audio and/or video data (for example videoblur) that can be used to represent experiential data stream contentthat is outside the content extents.

As discussed above, the extracted-experiential data stream can be storedin a format that allows playback by an existing DVD player (or itsequivalent). In addition, the extracted-experiential data stream can bestored in a format that can be used by a limited social televisionsystem. A limited social television system is one that has lesscapability than that for the social television system previouslydiscussed. One embodiment of a limited social television system is onethat is configured to include social television system playbackfunctions but does not include the capability of direct interaction withremote audience members. Examples of such playback functions are thosedescribed with respect to FIG. 14 and related to at least the‘synchronization command’ procedure 1413 and the ‘follow command’procedure 1417 as well as aspects of FIG. 16 at least related topresentation of commentary clips.

The limited social television system is a partial implementation of thepreviously described social television system. For example, oneembodiment does not provide support for direct interaction with remoteaudience members. Thus, the high level shared social space process 600is not needed (other than to start threads) as theextracted-experiential data stream is accessed from a provided media (orthrough the network). Further the ‘establish device communication’procedure 703 is not needed as the local audience member's limitedsocial television system is not in communication with a remote audiencemember's social television system. For similar reasons the limitedsocial television system does not need the initialization process 800,the ‘send local state’ procedure 905, the display thread process 1000,the first startup phase thread 1200, or the second phase startup thread1300 (although, the technology disclosed for these processes and threadscan still be useful. One skilled in the art will understand that thereexist many other embodiments of a limited social television system thatcan be based on the technology disclosed herein.

One skilled in the art will understand that the experiential data streamcan be an audio program, a video program, a multi-media program, or somecombination of these types of programs. Such a one will also understandthat the term replacement/additional content includes supplemental,replacement, additional, and/or commercial content.

One skilled in the art will understand that although the descriptionherein is cast within a multi threaded paradigm that the techniquesdisclosed are applicable to other programming paradigms. One skilled inthe art will understand that there are many different but equivalentways to start the threads just as there are many different butequivalent ways to architect the embodiments (including different dataflow and programming methodology, and the use of electronic circuitassists). Such a one will also understand that object-oriented,procedural programming, or other methodologies can also be used toimplement the technology disclosed within.

In addition, state changes can be detected by periodically comparing acopy of the state with the current state from information sent by theaudience members' devices over the control link. Another embodiment candetect a state-change event and respond to that event.

As used herein, a procedure is a self-consistent sequence of steps thatcan be performed by a programmed computer, specialized electronics orother circuitry or a combination thereof that lead to a desired result.These steps can be defined by one or more computer instructions. Thesesteps can be performed by a computer executing the instructions thatdefine the steps. Thus, the term “procedure” can refer (for example, butwithout limitation) to a sequence of instructions, a sequence ofinstructions organized within a programmed-procedure orprogrammed-function, a sequence of instructions organized withinprogrammed-processes executing in one or more computers, or a sequenceof steps performed by electronic or other circuitry; or other logic.Thus, for example, an extent expansion logic (or combination of logics)can be used to perform the methods for expanding the extents of thecontent to be extracted.

One skilled in the art will understand that the network transmitsinformation (such as data that defines a experiential data stream orcommentary audio clip as well as data that defines a computer program).The information can also be embodied within a carrier-wave. The term“carrier-wave” includes electromagnetic signals, visible or invisiblelight pulses, signals on a data bus, or signals transmitted over anywire, wireless, or optical fiber technology that allows information tobe transmitted over a network. Programs and data are commonly read fromboth tangible physical media (such as a compact, floppy, or magneticdisk) and from a network. Thus, the network, like a tangible physicalmedia, is a computer-usable data carrier.

The network 401 can be a broadcast network, a cable network, a satellitenetwork, the Internet, a LAN or WAN or some combination of these. Thevideo display component 413 can be a television set, a video monitor, acomputer, et cetera

One skilled in the art will understand that this disclosure teachesfundamental aspects of establishing a shared social space, of thedevices, methods, systems, products, and business methods to establish,maintain, and exploit the shared social space as well as techniques thatcan be used in a local social space.

From the foregoing, it will be appreciated that the disclosed technologyhas (without limitation) the following advantages:

1) The ability to extract and store portions of an experiential datastream and/or commentary from a social television system such that thestored portions can be presented to another to close the loop ofinteraction.

2) The ability to asynchronously create a social space by sharinghighlights of content from an experiential data stream and audiencemember commentary related to those highlights with another.

Although the presently claimed technology has been described in terms ofthe presently preferred embodiments, one skilled in the art willunderstand that various modifications and alterations may be madewithout departing from the scope of the claims. Accordingly, the scopeof the claims is not to be limited to the particular embodimentsdiscussed herein.

1. An apparatus, for presenting an extracted-experiential data stream,said apparatus: comprising: a media access logic configured to accesssaid extracted-experiential data stream; a first presentation logicconfigured to present content from said extracted-experiential datastream accessed by the media access logic; a position monitor logicconfigured to monitor a first playback position in a time dimension ofsaid extracted-experiential data stream; a proximity detector configuredto detect proximity of said first playback position to a commentary cliptrigger position associated with a commentary clip; and a secondpresentation logic configured to present the commentary clip inconjunction with the first presentation logic.
 2. The apparatus of claim1, further comprising: a communication mechanism configured to receive acommand request from a social-space related user selectable control; andan internal control handler configured to process said command requestto perform an operation related to the extracted-experiential datastream.
 3. The apparatus of claim 1, wherein the proximity detector isfurther configured to detect proximity of said first playback positionto an insertion coordinate associated with replacement/additionalcontent; and the second presentation logic is further configured topresent said replacement/additional content at said insertion coordinateinstead of said extracted-experiential data stream content.
 4. Theapparatus of claim 1, wherein the proximity detector is furtherconfigured to detect proximity of said first playback position to aninsertion coordinate associated with replacement/additional content; andthe second presentation logic is further configured to present saidreplacement/additional content at said insertion coordinate instead ofsaid extracted-experiential data stream content.
 5. The apparatus ofclaim 4, further comprising: a completion logic configured to detectcompletion of the presentation of said replacement/additional content;and a resumption logic, responsive to the completion logic, configuredto resume presentation of said extracted-experiential data streamcontent at a resumption coordinate within said extracted-experientialdata stream.
 6. The apparatus of claim 1, wherein said commentary clipis one of a plurality of commentary clips each having a respective arespective commentary clip trigger position and wherein some of saidplurality of commentary clips are associated with a group.
 7. Theapparatus of claim 6, wherein said some of said plurality of commentaryclips are enabled for presentation as said group.